elasticsearch_cluster

ElasticsearchCluster module.

exception spicerack.elasticsearch_cluster.ElasticsearchClusterCheckError[source]

Bases: SpicerackCheckError

Custom Exception class for check errors of this module.

exception spicerack.elasticsearch_cluster.ElasticsearchClusterError[source]

Bases: SpicerackError

Custom Exception class for errors of this module.

class spicerack.elasticsearch_cluster.ElasticsearchCluster(elasticsearch: Elasticsearch, remote: Remote, dry_run: bool = True)[source]

Bases: object

Class to manage elasticsearch cluster.

Initialize ElasticsearchCluster.

Parameters:

  • elasticsearch (elasticsearch.Elasticsearch) -- elasticsearch instance.

  • remote (spicerack.remote.Remote) -- the Remote instance.

  • dry_run (bool, optional) -- whether this is a DRY-RUN.

check_green() → None[source]

Cluster health status.

Raises:

spicerack.elasticsearch_cluster.ElasticsearchClusterCheckError -- This is raised when request times and cluster is not green.

check_yellow_w_no_moving_shards() → None[source]

Cluster health status.

Raises:

spicerack.elasticsearch_cluster.ElasticsearchClusterCheckError -- This is raised when request times and cluster is not yellow with no initializing or relocating shards.

flush_markers(timeout: timedelta = datetime.timedelta(seconds=60)) → None[source]

Flush markers unsynced.

Note

flush and flush_synced are called here because from experience, it results in fewer shards not syncing. This also makes the recovery faster.

Parameters:

timeout (datetime.timedelta) -- timedelta object for elasticsearch request timeout.

force_allocation_of_all_unassigned_shards() → None[source]

Manual allocation of unassigned shards.

frozen_writes(reason: Reason) → Iterator[None][source]

Stop writes to all elasticsearch indices and enable them on exit.

Parameters:

reason (spicerack.administrative.Reason) -- Reason for freezing writes.

get_nodes() → Dict[source]

Get all Elasticsearch Nodes.

Returns:

dictionary of elasticsearch nodes in the cluster.

Return type:

dict

is_node_in_cluster_nodes(node: str) → bool[source]

Checks if node is in a list of elasticsearch cluster nodes.

Parameters:

node (str) -- the elasticsearch host.

Returns:

True if node is present and False if not.

Return type:

bool

reset_indices_to_read_write() → None[source]

Reset all readonly indices to read/write.

In some cases (running low on disk space), indices are switched to readonly. This method will update all readonly indices to read/write.

stopped_replication() → Iterator[None][source]

Context manager to perform actions while the cluster replication is stopped.

class spicerack.elasticsearch_cluster.ElasticsearchClusters(clusters: Sequence[ElasticsearchCluster], remote: Remote, prometheus: Prometheus, write_queue_datacenters: Sequence[str], dry_run: bool = True)[source]

Bases: object

Class to manage elasticsearch clusters.

Initialize ElasticsearchClusters.

Parameters:
flush_markers(timeout: timedelta = datetime.timedelta(seconds=60)) → None[source]

Flush markers on all clusters.

Parameters:

timeout (datetime.timedelta, optional) -- timedelta object for elasticsearch request timeout.

force_allocation_of_all_unassigned_shards() → None[source]

Force allocation of unassigned shards on all clusters.

frozen_writes(reason: Reason) → Iterator[List[None]][source]

Freeze all writes to the clusters and then perform operations before unfreezing writes.

Parameters:

reason (spicerack.administrative.Reason) -- Reason for freezing writes.

Yields:

list -- a side-effect list of None, as a result of the stack of context managers.

get_next_clusters_nodes(started_before: datetime, size: int = 1) → ElasticsearchHosts | None[source]

Get next set of cluster nodes for cookbook operations like upgrade, rolling restart etc.

Nodes are selected from the row with the least restarted nodes. This ensures that a row is fully upgraded before moving to the next row. Since shards cannot move to a node with an older version of elasticsearch, this should help to keep all shards allocated at all times.

Master capable nodes are returned after all other nodes have restarted to support version upgrades which strongly suggest the masters are upgraded last.

Parameters:

  • started_before (datetime.datetime) -- the time against after which we check if the node has been restarted.

  • size (int, optional) -- size of nodes not restarted in a row.

Returns:

next eligible nodes for ElasticsearchHosts or

None when all nodes have been processed.

Return type:

spicerack.elasticsearch_cluster.ElasticsearchHosts

reset_indices_to_read_write() → None[source]

Reset all readonly indices to read/write.

In some cases (running low on disk space), indices are switched to readonly. This method will update all readonly indices to read/write.

stopped_replication() → Iterator[List[None]][source]

Stops replication for all clusters.

Yields:

list -- a side-effect list of None, as a result of the stack of context managers.

wait_for_all_write_queues_empty() → None[source]

Wait for all relevant CirrusSearch write queues to be empty.

Checks the Prometheus server in each of the CORE_DATACENTERS

At most waits for 60*60 seconds = 1 hour.

Does not retry if prometheus returns empty results for all datacenters.

wait_for_green(timeout: timedelta = datetime.timedelta(seconds=3600)) → None[source]

Wait for green on all clusters.

Parameters:

timeout (datetime.timedelta, optional) -- timedelta object to represent how long to wait for green status on all clusters.

wait_for_yellow_w_no_moving_shards(timeout: timedelta = datetime.timedelta(seconds=3600)) → None[source]

Wait for a yellow cluster status with no relocating or initializing shards.

Parameters:

timeout (datetime.timedelta, optional) -- timedelta object to represent how long to wait for no yellow status with no initializing or relocating shards on all clusters.

class spicerack.elasticsearch_cluster.ElasticsearchHosts(remote_hosts: RemoteHosts, nodes: Sequence[NodesGroup], dry_run: bool = True)[source]

Bases: RemoteHostsAdapter

Remotehosts Adapter for managing elasticsearch nodes.

After calling the super's constructor, initialize other instance variables.

Parameters:

  • remote_hosts (spicerack.remote.RemoteHosts) -- the instance with the target hosts.

  • nodes (list) -- list of dicts containing clusters hosts belong to.

  • dry_run (bool, optional) -- whether this is a DRY-RUN.

depool_nodes() → None[source]

Depool the hosts.

get_remote_hosts() → RemoteHosts[source]

Returns elasticsearch remote hosts.

Returns:

RemoteHosts instance for this adapter.

Return type:

spicerack.remote.RemoteHosts

pool_nodes() → None[source]

Pool the hosts.

restart_elasticsearch() → None[source]

Restarts all elasticsearch instances.

start_elasticsearch() → None[source]

Starts all elasticsearch instances.

stop_elasticsearch() → None[source]

Stops all elasticsearch instances.

wait_for_elasticsearch_up(timeout: timedelta = datetime.timedelta(seconds=900)) → None[source]

Check if elasticsearch instances on each node are up.

Parameters:

timeout (datetime.timedelta, optional) -- represent how long to wait for all instances to be up.

class spicerack.elasticsearch_cluster.NodesGroup(json_node: Dict, cluster: ElasticsearchCluster)[source]

Bases: object

Internal class, used for parsing responses from the elasticsearch node API.

Since the same server can host multiple elasticsearch instances, this class can consolidate those multiple instances in a single object.

Instantiate a new node.

Parameters:
accumulate(json_node: Dict, cluster: ElasticsearchCluster) → None[source]

Accumulate information from other elasticsearch instances running on the same server.

Parameters:

  • json_node (dict) -- a single node, as returned from the elasticsearch API.

  • cluster (elasticsearch.Elasticsearch) -- an elasticsearch instance

check_all_nodes_up() → None[source]

Check that all the nodes on this hosts are up and have joined their respective clusters.

Raises:

spicerack.elasticsearch_cluster.ElasticsearchClusterCheckError -- if not all nodes have joined.

restarted_since(since: datetime) → bool[source]

Check if node has been restarted.

Parameters:

since (datetime.datetime) -- the time against after which we check if the node has been restarted.

Returns:

True if the node has been restarted after since, false otherwise.

Return type:

bool

property clusters_instances: Sequence[ElasticsearchCluster]

Cluster instances running on this node group.

property fqdn: str

Fully Qualified Domain Name.

property master_capable: Set[str]

Set of clusters this node is master capable on.

property row: str

Datacenter row.

spicerack.elasticsearch_cluster.create_elasticsearch_clusters(configuration: Dict[str, Dict[str, Dict[str, str]]], clustergroup: str, write_queue_datacenters: Sequence[str], remote: Remote, prometheus: Prometheus, dry_run: bool = True) → ElasticsearchClusters[source]

Create ElasticsearchClusters instance.

Parameters:

  • clustergroup (str) -- name of cluster group.

  • write_queue_datacenters (Sequence[str]) -- Sequence of which core DCs to query write queues for.

  • remote (spicerack.remote.Remote) -- the Remote instance.

  • prometheus (wmflib.prometheus.Prometheus) -- the prometheus instance.

  • dry_run (bool, optional) -- whether this is a DRY-RUN.

Raises:

spicerack.elasticsearch_cluster.ElasticsearchClusterError -- Thrown when the requested cluster configuration is not found.

Returns:

ElasticsearchClusters instance.

Return type:

spicerack.elasticsearch_cluster.ElasticsearchClusters