elasticsearch_cluster¶
ElasticsearchCluster module.
- exception spicerack.elasticsearch_cluster.ElasticsearchClusterCheckError[source]¶
Bases:
spicerack.exceptions.SpicerackCheckError
Custom Exception class for check errors of this module.
- exception spicerack.elasticsearch_cluster.ElasticsearchClusterError[source]¶
Bases:
spicerack.exceptions.SpicerackError
Custom Exception class for errors of this module.
- class spicerack.elasticsearch_cluster.ElasticsearchCluster(elasticsearch: elasticsearch.client.Elasticsearch, remote: spicerack.remote.Remote, dry_run: bool = True)[source]¶
Bases:
object
Class to manage elasticsearch cluster.
Initialize ElasticsearchCluster.
- Parameters
elasticsearch (elasticsearch.Elasticsearch) -- elasticsearch instance.
remote (spicerack.remote.Remote) -- the Remote instance.
dry_run (bool, optional) -- whether this is a DRY-RUN.
- check_green() None [source]¶
Cluster health status.
- Raises
spicerack.elasticsearch_cluster.ElasticsearchClusterCheckError -- This is raised when request times and cluster is not green.
- check_yellow_w_no_moving_shards() None [source]¶
Cluster health status.
- Raises
spicerack.elasticsearch_cluster.ElasticsearchClusterCheckError -- This is raised when request times and cluster is not yellow with no initializing or relocating shards.
- flush_markers(timeout: datetime.timedelta = datetime.timedelta(seconds=60)) None [source]¶
Flush markers unsynced.
Note
flush
andflush_synced
are called here because from experience, it results in fewer shards not syncing. This also makes the recovery faster.- Parameters
timeout (datetime.timedelta) -- timedelta object for elasticsearch request timeout.
- frozen_writes(reason: spicerack.administrative.Reason) Iterator[None] [source]¶
Stop writes to all elasticsearch indices and enable them on exit.
- Parameters
reason (spicerack.administrative.Reason) -- Reason for freezing writes.
- get_nodes() Dict [source]¶
Get all Elasticsearch Nodes.
- Returns
dictionary of elasticsearch nodes in the cluster.
- Return type
- is_node_in_cluster_nodes(node: str) bool [source]¶
Checks if node is in a list of elasticsearch cluster nodes.
- class spicerack.elasticsearch_cluster.ElasticsearchClusters(clusters: Sequence[spicerack.elasticsearch_cluster.ElasticsearchCluster], remote: spicerack.remote.Remote, prometheus: wmflib.prometheus.Prometheus, write_queue_datacenters: Sequence[str], dry_run: bool = True)[source]¶
Bases:
object
Class to manage elasticsearch clusters.
Initialize ElasticsearchClusters.
- Parameters
clusters (list) -- list of
spicerack.elasticsearch_cluster.ElasticsearchCluster
instances.remote (spicerack.remote.Remote) -- the Remote instance.
prometheus (wmflib.prometheus.Prometheus) -- the prometheus instance.
write_queue_datacenters (Sequence[str]) -- Sequence of which core DCs to query write queues for.
dry_run (bool, optional) -- whether this is a DRY-RUN.
- flush_markers(timeout: datetime.timedelta = datetime.timedelta(seconds=60)) None [source]¶
Flush markers on all clusters.
- Parameters
timeout (datetime.timedelta, optional) -- timedelta object for elasticsearch request timeout.
- force_allocation_of_all_unassigned_shards() None [source]¶
Force allocation of unassigned shards on all clusters.
- frozen_writes(reason: spicerack.administrative.Reason) Iterator[List[None]] [source]¶
Freeze all writes to the clusters and then perform operations before unfreezing writes.
- Parameters
reason (spicerack.administrative.Reason) -- Reason for freezing writes.
- Yields
list -- a side-effect list of
None
, as a result of the stack of context managers.
- get_next_clusters_nodes(started_before: datetime.datetime, size: int = 1) Optional[spicerack.elasticsearch_cluster.ElasticsearchHosts] [source]¶
Get next set of cluster nodes for cookbook operations like upgrade, rolling restart etc.
Nodes are selected from the row with the least restarted nodes. This ensures that a row is fully upgraded before moving to the next row. Since shards cannot move to a node with an older version of elasticsearch, this should help to keep all shards allocated at all times.
Master capable nodes are returned after all other nodes have restarted to support version upgrades which strongly suggest the masters are upgraded last.
- Parameters
started_before (datetime.datetime) -- the time against after which we check if the node has been restarted.
size (int, optional) -- size of nodes not restarted in a row.
- Returns
- next eligible nodes for ElasticsearchHosts or
None
when all nodes have been processed.
- Return type
- reset_indices_to_read_write() None [source]¶
Reset all readonly indices to read/write.
In some cases (running low on disk space), indices are switched to readonly. This method will update all readonly indices to read/write.
- stopped_replication() Iterator[List[None]] [source]¶
Stops replication for all clusters.
- Yields
list -- a side-effect list of
None
, as a result of the stack of context managers.
- wait_for_all_write_queues_empty() None [source]¶
Wait for all relevant CirrusSearch write queues to be empty.
Checks the Prometheus server in each of the CORE_DATACENTERS
At most waits for 60*60 seconds = 1 hour.
Does not retry if prometheus returns empty results for all datacenters.
- wait_for_green(timeout: datetime.timedelta = datetime.timedelta(seconds=3600)) None [source]¶
Wait for green on all clusters.
- Parameters
timeout (datetime.timedelta, optional) -- timedelta object to represent how long to wait for green status on all clusters.
- wait_for_yellow_w_no_moving_shards(timeout: datetime.timedelta = datetime.timedelta(seconds=3600)) None [source]¶
Wait for a yellow cluster status with no relocating or initializing shards.
- Parameters
timeout (datetime.timedelta, optional) -- timedelta object to represent how long to wait for no yellow status with no initializing or relocating shards on all clusters.
- class spicerack.elasticsearch_cluster.ElasticsearchHosts(remote_hosts: spicerack.remote.RemoteHosts, nodes: Sequence[spicerack.elasticsearch_cluster.NodesGroup], dry_run: bool = True)[source]¶
Bases:
spicerack.remote.RemoteHostsAdapter
Remotehosts Adapter for managing elasticsearch nodes.
After calling the super's constructor, initialize other instance variables.
- Parameters
remote_hosts (spicerack.remote.RemoteHosts) -- the instance with the target hosts.
nodes (list) -- list of dicts containing clusters hosts belong to.
dry_run (bool, optional) -- whether this is a DRY-RUN.
- get_remote_hosts() spicerack.remote.RemoteHosts [source]¶
Returns elasticsearch remote hosts.
- Returns
RemoteHosts instance for this adapter.
- Return type
- wait_for_elasticsearch_up(timeout: datetime.timedelta = datetime.timedelta(seconds=900)) None [source]¶
Check if elasticsearch instances on each node are up.
- Parameters
timeout (datetime.timedelta, optional) -- represent how long to wait for all instances to be up.
- class spicerack.elasticsearch_cluster.NodesGroup(json_node: Dict, cluster: spicerack.elasticsearch_cluster.ElasticsearchCluster)[source]¶
Bases:
object
Internal class, used for parsing responses from the elasticsearch node API.
Since the same server can host multiple elasticsearch instances, this class can consolidate those multiple instances in a single object.
Instantiate a new node.
- Parameters
json_node (dict) -- a single node, as returned from the elasticsearch API.
cluster (spicerack.elasticsearch_cluster.ElasticsearchCluster) -- an elasticsearch instance
- accumulate(json_node: Dict, cluster: spicerack.elasticsearch_cluster.ElasticsearchCluster) None [source]¶
Accumulate information from other elasticsearch instances running on the same server.
- Parameters
json_node (dict) -- a single node, as returned from the elasticsearch API.
cluster (elasticsearch.Elasticsearch) -- an elasticsearch instance
- check_all_nodes_up() None [source]¶
Check that all the nodes on this hosts are up and have joined their respective clusters.
- Raises
spicerack.elasticsearch_cluster.ElasticsearchClusterCheckError -- if not all nodes have joined.
- restarted_since(since: datetime.datetime) bool [source]¶
Check if node has been restarted.
- Parameters
since (datetime.datetime) -- the time against after which we check if the node has been restarted.
- Returns
True if the node has been restarted after since, false otherwise.
- Return type
- property clusters_instances: Sequence[spicerack.elasticsearch_cluster.ElasticsearchCluster]¶
Cluster instances running on this node group.
- spicerack.elasticsearch_cluster.create_elasticsearch_clusters(configuration: Dict[str, Dict[str, Dict[str, str]]], clustergroup: str, write_queue_datacenters: Sequence[str], remote: spicerack.remote.Remote, prometheus: wmflib.prometheus.Prometheus, dry_run: bool = True) spicerack.elasticsearch_cluster.ElasticsearchClusters [source]¶
Create ElasticsearchClusters instance.
- Parameters
clustergroup (str) -- name of cluster group.
write_queue_datacenters (Sequence[str]) -- Sequence of which core DCs to query write queues for.
remote (spicerack.remote.Remote) -- the Remote instance.
prometheus (wmflib.prometheus.Prometheus) -- the prometheus instance.
dry_run (bool, optional) -- whether this is a DRY-RUN.
- Raises
spicerack.elasticsearch_cluster.ElasticsearchClusterError -- Thrown when the requested cluster configuration is not found.
- Returns
ElasticsearchClusters instance.
- Return type