elasticsearch_cluster

ElasticsearchCluster module.

class spicerack.elasticsearch_cluster.ElasticsearchCluster(elasticsearch: elasticsearch.client.Elasticsearch, remote: spicerack.remote.Remote, dry_run: bool = True)[source]

Bases: object

Class to manage elasticsearch cluster.

Initialize ElasticsearchCluster.

Parameters
  • elasticsearch (elasticsearch.Elasticsearch) -- elasticsearch instance.

  • remote (spicerack.remote.Remote) -- the Remote instance.

  • dry_run (bool, optional) -- whether this is a DRY-RUN.

check_green() → None[source]

Cluster health status.

Raises

spicerack.elasticsearch_cluster.ElasticsearchClusterCheckError -- This is raised when request times and cluster is not green.

flush_markers(timeout: datetime.timedelta = datetime.timedelta(0, 60)) → None[source]

Flush markers unsynced.

Note

flush and flush_synced are called here because from experience, it results in fewer shards not syncing. This also makes the recovery faster.

Parameters

timeout (datetime.timedelta) -- timedelta object for elasticsearch request timeout.

force_allocation_of_all_unassigned_shards() → None[source]

Manual allocation of unassigned shards.

frozen_writes(reason: spicerack.administrative.Reason) → Iterator[None][source]

Stop writes to all elasticsearch indices and enable them on exit.

Parameters

reason (spicerack.administrative.Reason) -- Reason for freezing writes.

get_nodes() → Dict[source]

Get all Elasticsearch Nodes.

Returns

dictionary of elasticsearch nodes in the cluster.

Return type

dict

is_node_in_cluster_nodes(node: str) → bool[source]

Checks if node is in a list of elasticsearch cluster nodes.

Parameters

node (str) -- the elasticsearch host.

Returns

True if node is present and False if not.

Return type

bool

reset_indices_to_read_write() → None[source]

Reset all readonly indices to read/write.

In some cases (running low on disk space), indices are switched to readonly. This method will update all readonly indices to read/write.

stopped_replication() → Iterator[None][source]

Context manager to perform actions while the cluster replication is stopped.

exception spicerack.elasticsearch_cluster.ElasticsearchClusterCheckError[source]

Bases: spicerack.exceptions.SpicerackCheckError

Custom Exception class for check errors of this module.

exception spicerack.elasticsearch_cluster.ElasticsearchClusterError[source]

Bases: spicerack.exceptions.SpicerackError

Custom Exception class for errors of this module.

class spicerack.elasticsearch_cluster.ElasticsearchClusters(clusters: Sequence[ElasticsearchCluster], remote: spicerack.remote.Remote, dry_run: bool = True)[source]

Bases: object

Class to manage elasticsearch clusters.

Initialize ElasticsearchClusters.

Parameters
flush_markers(timeout: datetime.timedelta = datetime.timedelta(0, 60)) → None[source]

Flush markers on all clusters.

Parameters

timeout (datetime.timedelta, optional) -- timedelta object for elasticsearch request timeout.

force_allocation_of_all_unassigned_shards() → None[source]

Force allocation of unassigned shards on all clusters.

frozen_writes(reason: spicerack.administrative.Reason) → Iterator[List[None]][source]

Freeze all writes to the clusters and then perform operations before unfreezing writes.

Parameters

reason (spicerack.administrative.Reason) -- Reason for freezing writes.

Yields

list -- a side-effect list of None, as a result of the stack of context managers.

get_next_clusters_nodes(started_before: datetime.datetime, size: int = 1) → Optional[spicerack.elasticsearch_cluster.ElasticsearchHosts][source]

Get next set of cluster nodes for cookbook operations like upgrade, rolling restart etc.

Nodes are selected from the row with the least restarted nodes. This ensure that a row is fully upgraded before moving to the next row. Since shards cannot move to a node with an older version of elasticsearch, this should help to keep all shards allocated at all time.

Parameters
  • started_before (datetime.datetime) -- the time against after which we check if the node has been restarted.

  • size (int, optional) -- size of nodes not restarted in a row.

Returns

next eligible nodes for ElasticsearchHosts or

None when all nodes have been processed.

Return type

spicerack.elasticsearch_cluster.ElasticsearchHosts

reset_indices_to_read_write() → None[source]

Reset all readonly indices to read/write.

In some cases (running low on disk space), indices are switched to readonly. This method will update all readonly indices to read/write.

stopped_replication() → Iterator[List[None]][source]

Stops replication for all clusters.

Yields

list -- a side-effect list of None, as a result of the stack of context managers.

wait_for_green(timeout: datetime.timedelta = datetime.timedelta(0, 3600)) → None[source]

Wait for green on all clusters.

Parameters

timeout (datetime.timedelta, optional) -- timedelta object to represent how long to wait for green status on all clusters.

class spicerack.elasticsearch_cluster.ElasticsearchHosts(remote_hosts: spicerack.remote.RemoteHosts, nodes: Sequence[NodesGroup], dry_run: bool = True)[source]

Bases: spicerack.remote.RemoteHostsAdapter

Remotehosts Adapter for managing elasticsearch nodes.

After calling the super's constructor, initialize other instance variables.

Parameters
  • remote_hosts (spicerack.remote.RemoteHosts) -- the instance with the target hosts.

  • nodes (list) -- list of dicts containing clusters hosts belong to.

  • dry_run (bool, optional) -- whether this is a DRY-RUN.

depool_nodes() → None[source]

Depool the hosts.

get_remote_hosts() → spicerack.remote.RemoteHosts[source]

Returns elasticsearch remote hosts.

Returns

RemoteHosts instance for this adapter.

Return type

spicerack.remote.RemoteHosts

pool_nodes() → None[source]

Pool the hosts.

restart_elasticsearch() → None[source]

Restarts all elasticsearch instances.

start_elasticsearch() → None[source]

Starts all elasticsearch instances.

stop_elasticsearch() → None[source]

Stops all elasticsearch instances.

wait_for_elasticsearch_up(timeout: datetime.timedelta = datetime.timedelta(0, 900)) → None[source]

Check if elasticsearch instances on each node are up.

Parameters

timeout (datetime.timedelta, optional) -- represent how long to wait for all instances to be up.

class spicerack.elasticsearch_cluster.NodesGroup(json_node: Dict, cluster: spicerack.elasticsearch_cluster.ElasticsearchCluster)[source]

Bases: object

Internal class, used for parsing responses from the elasticsearch node API.

Since the same server can host multiple elasticsearch instances, this class can consolidate those multiple instances in a single object.

Instantiate a new node.

Parameters
accumulate(json_node: Dict, cluster: spicerack.elasticsearch_cluster.ElasticsearchCluster) → None[source]

Accumulate information from other elasticsearch instances running on the same server.

Parameters
  • json_node (dict) -- a single node, as returned from the elasticsearch API.

  • cluster (elasticsearch.Elasticsearch) -- an elasticsearch instance

check_all_nodes_up() → None[source]

Check that all the nodes on this hosts are up and have joined their respective clusters.

Raises

spicerack.elasticsearch_cluster.ElasticsearchClusterCheckError -- if not all nodes have joined.

property clusters_instances

Cluster instances running on this node group.

property fqdn

Fully Qualified Domain Name.

restarted_since(since: datetime.datetime) → bool[source]

Check if node has been restarted.

Parameters

since (datetime.datetime) -- the time against after which we check if the node has been restarted.

Returns

True if the node has been restarted after since, false otherwise.

Return type

bool

property row

Datacenter row.

spicerack.elasticsearch_cluster.create_elasticsearch_clusters(clustergroup: str, remote: spicerack.remote.Remote, dry_run: bool = True) → spicerack.elasticsearch_cluster.ElasticsearchClusters[source]

Create ElasticsearchClusters instance.

Parameters
  • clustergroup (str) -- name of cluster group.

  • remote (spicerack.remote.Remote) -- the Remote instance.

  • dry_run (bool, optional) -- whether this is a DRY-RUN.

Raises

spicerack.elasticsearch_cluster.ElasticsearchClusterError -- Thrown when the requested cluster configuration is not found.

Returns

ElasticsearchClusters instance.

Return type

spicerack.elasticsearch_cluster.ElasticsearchClusters