elasticsearch_cluster¶

ElasticsearchCluster module.

exception spicerack.elasticsearch_cluster.ElasticsearchClusterCheckError[source]¶

Bases: spicerack.exceptions.SpicerackCheckError

Custom Exception class for check errors of this module.

exception spicerack.elasticsearch_cluster.ElasticsearchClusterError[source]¶

Bases: spicerack.exceptions.SpicerackError

Custom Exception class for errors of this module.

class spicerack.elasticsearch_cluster.ElasticsearchCluster(elasticsearch: elasticsearch.client.Elasticsearch, remote: spicerack.remote.Remote, dry_run: bool = True)[source]¶

Bases: object

Class to manage elasticsearch cluster.

Initialize ElasticsearchCluster.

Parameters

elasticsearch (elasticsearch.Elasticsearch) -- elasticsearch instance.
remote (spicerack.remote.Remote) -- the Remote instance.
dry_run (bool, optional) -- whether this is a DRY-RUN.

check_green() → None[source]¶

Cluster health status.

Raises: spicerack.elasticsearch_cluster.ElasticsearchClusterCheckError -- This is raised when request times and cluster is not green.

flush_markers(timeout: datetime.timedelta = datetime.timedelta(seconds=60)) → None[source]¶

Flush markers unsynced.

Note

flush and flush_synced are called here because from experience, it results in fewer shards not syncing. This also makes the recovery faster.

Parameters: timeout (datetime.timedelta) -- timedelta object for elasticsearch request timeout.

force_allocation_of_all_unassigned_shards() → None[source]¶: Manual allocation of unassigned shards.

frozen_writes(reason: spicerack.administrative.Reason) → Iterator[None][source]¶

Stop writes to all elasticsearch indices and enable them on exit.

Parameters: reason (spicerack.administrative.Reason) -- Reason for freezing writes.

get_nodes() → Dict[source]¶

Get all Elasticsearch Nodes.

Returns: dictionary of elasticsearch nodes in the cluster.
Return type: dict

is_node_in_cluster_nodes(node: str) → bool[source]¶

Checks if node is in a list of elasticsearch cluster nodes.

Parameters: node (str) -- the elasticsearch host.
Returns: True if node is present and False if not.
Return type: bool

reset_indices_to_read_write() → None[source]¶

Reset all readonly indices to read/write.

In some cases (running low on disk space), indices are switched to readonly. This method will update all readonly indices to read/write.

stopped_replication() → Iterator[None][source]¶: Context manager to perform actions while the cluster replication is stopped.

class spicerack.elasticsearch_cluster.ElasticsearchClusters(clusters: Sequence[spicerack.elasticsearch_cluster.ElasticsearchCluster], remote: spicerack.remote.Remote, prometheus: wmflib.prometheus.Prometheus, write_queue_datacenters: Sequence[str], dry_run: bool = True)[source]¶

Bases: object

Class to manage elasticsearch clusters.

Initialize ElasticsearchClusters.

Parameters

clusters (list) -- list of spicerack.elasticsearch_cluster.ElasticsearchCluster instances.
remote (spicerack.remote.Remote) -- the Remote instance.
prometheus (wmflib.prometheus.Prometheus) -- the prometheus instance.
write_queue_datacenters (Sequence[str]) -- Sequence of which core DCs to query write queues for.
dry_run (bool, optional) -- whether this is a DRY-RUN.

flush_markers(timeout: datetime.timedelta = datetime.timedelta(seconds=60)) → None[source]¶

Flush markers on all clusters.

Parameters: timeout (datetime.timedelta, optional) -- timedelta object for elasticsearch request timeout.

force_allocation_of_all_unassigned_shards() → None[source]¶: Force allocation of unassigned shards on all clusters.

frozen_writes(reason: spicerack.administrative.Reason) → Iterator[List[None]][source]¶

Freeze all writes to the clusters and then perform operations before unfreezing writes.

Parameters: reason (spicerack.administrative.Reason) -- Reason for freezing writes.
Yields: list -- a side-effect list of None, as a result of the stack of context managers.

get_next_clusters_nodes(started_before: datetime.datetime, size: int = 1) → Optional[spicerack.elasticsearch_cluster.ElasticsearchHosts][source]¶

Get next set of cluster nodes for cookbook operations like upgrade, rolling restart etc.

Nodes are selected from the row with the least restarted nodes. This ensures that a row is fully upgraded before moving to the next row. Since shards cannot move to a node with an older version of elasticsearch, this should help to keep all shards allocated at all times.

Parameters

started_before (datetime.datetime) -- the time against after which we check if the node has been restarted.
size (int, optional) -- size of nodes not restarted in a row.

Returns

next eligible nodes for ElasticsearchHosts or: None when all nodes have been processed.

Return type

spicerack.elasticsearch_cluster.ElasticsearchHosts

reset_indices_to_read_write() → None[source]¶

Reset all readonly indices to read/write.

In some cases (running low on disk space), indices are switched to readonly. This method will update all readonly indices to read/write.

stopped_replication() → Iterator[List[None]][source]¶

Stops replication for all clusters.

Yields: list -- a side-effect list of None, as a result of the stack of context managers.

wait_for_all_write_queues_empty() → None[source]¶

Wait for all relevant CirrusSearch write queues to be empty.

Checks the Prometheus server in each of the CORE_DATACENTERS

At most waits for 60*60 seconds = 1 hour.

Does not retry if prometheus returns empty results for all datacenters.

wait_for_green(timeout: datetime.timedelta = datetime.timedelta(seconds=3600)) → None[source]¶

Wait for green on all clusters.

Parameters: timeout (datetime.timedelta, optional) -- timedelta object to represent how long to wait for green status on all clusters.

class spicerack.elasticsearch_cluster.ElasticsearchHosts(remote_hosts: spicerack.remote.RemoteHosts, nodes: Sequence[spicerack.elasticsearch_cluster.NodesGroup], dry_run: bool = True)[source]¶

Bases: spicerack.remote.RemoteHostsAdapter

Remotehosts Adapter for managing elasticsearch nodes.

After calling the super's constructor, initialize other instance variables.

Parameters

remote_hosts (spicerack.remote.RemoteHosts) -- the instance with the target hosts.
nodes (list) -- list of dicts containing clusters hosts belong to.
dry_run (bool, optional) -- whether this is a DRY-RUN.

depool_nodes() → None[source]¶: Depool the hosts.

get_remote_hosts() → spicerack.remote.RemoteHosts[source]¶

Returns elasticsearch remote hosts.

Returns: RemoteHosts instance for this adapter.
Return type: spicerack.remote.RemoteHosts

pool_nodes() → None[source]¶: Pool the hosts.

restart_elasticsearch() → None[source]¶: Restarts all elasticsearch instances.

start_elasticsearch() → None[source]¶: Starts all elasticsearch instances.

stop_elasticsearch() → None[source]¶: Stops all elasticsearch instances.

wait_for_elasticsearch_up(timeout: datetime.timedelta = datetime.timedelta(seconds=900)) → None[source]¶

Check if elasticsearch instances on each node are up.

Parameters: timeout (datetime.timedelta, optional) -- represent how long to wait for all instances to be up.

class spicerack.elasticsearch_cluster.NodesGroup(json_node: Dict, cluster: spicerack.elasticsearch_cluster.ElasticsearchCluster)[source]¶

Bases: object

Internal class, used for parsing responses from the elasticsearch node API.

Since the same server can host multiple elasticsearch instances, this class can consolidate those multiple instances in a single object.

Instantiate a new node.

Parameters

json_node (dict) -- a single node, as returned from the elasticsearch API.
cluster (spicerack.elasticsearch_cluster.ElasticsearchCluster) -- an elasticsearch instance

accumulate(json_node: Dict, cluster: spicerack.elasticsearch_cluster.ElasticsearchCluster) → None[source]¶

Accumulate information from other elasticsearch instances running on the same server.

Parameters

json_node (dict) -- a single node, as returned from the elasticsearch API.
cluster (elasticsearch.Elasticsearch) -- an elasticsearch instance

check_all_nodes_up() → None[source]¶

Check that all the nodes on this hosts are up and have joined their respective clusters.

Raises: spicerack.elasticsearch_cluster.ElasticsearchClusterCheckError -- if not all nodes have joined.

restarted_since(since: datetime.datetime) → bool[source]¶

Check if node has been restarted.

Parameters: since (datetime.datetime) -- the time against after which we check if the node has been restarted.
Returns: True if the node has been restarted after since, false otherwise.
Return type: bool

property clusters_instances: Sequence[spicerack.elasticsearch_cluster.ElasticsearchCluster]¶: Cluster instances running on this node group.

property fqdn: str¶: Fully Qualified Domain Name.

property row: str¶: Datacenter row.

spicerack.elasticsearch_cluster.create_elasticsearch_clusters(configuration: Dict[str, Dict[str, Dict[str, str]]], clustergroup: str, write_queue_datacenters: Sequence[str], remote: spicerack.remote.Remote, prometheus: wmflib.prometheus.Prometheus, dry_run: bool = True) → spicerack.elasticsearch_cluster.ElasticsearchClusters[source]¶

Create ElasticsearchClusters instance.

Parameters

clustergroup (str) -- name of cluster group.
write_queue_datacenters (Sequence[str]) -- Sequence of which core DCs to query write queues for.
remote (spicerack.remote.Remote) -- the Remote instance.
prometheus (wmflib.prometheus.Prometheus) -- the prometheus instance.
dry_run (bool, optional) -- whether this is a DRY-RUN.

Raises

spicerack.elasticsearch_cluster.ElasticsearchClusterError -- Thrown when the requested cluster configuration is not found.

Returns

ElasticsearchClusters instance.

Return type

spicerack.elasticsearch_cluster.ElasticsearchClusters