mysql

MySQL module.

TODO: replace with a proper MySQL module that uses a Python MySQL client, preferably in a parallel way.

spicerack.mysql.CORE_SECTIONS = ('s1', 's2', 's3', 's4', 's5', 's6', 's7', 's8', 'x1', 'es2', 'es3')

list of valid MySQL section names.

Type:tuple
class spicerack.mysql.Mysql(remote, dry_run=True)[source]

Bases: object

Class to manage MySQL servers.

__init__(remote, dry_run=True)[source]

Initialize the instance.

Parameters:
_check_core_master_in_sync(datacenter, heartbeat_dc, section, parent_heartbeat)[source]

Check and retry that the heartbeat value in a core DB master in DC is in sync with the provided heartbeat.

Parameters:
  • datacenter (str) -- the name of the datacenter from where to get the heartbeat value.
  • heartbeat_dc (str) -- the name of the datacenter for which to filter the heartbeat query.
  • section (str) -- the section name from where to get the heartbeat value and filter the heartbeat query.
  • master_heartbeat (datetime.datetime) -- the reference heartbeat from the parent master to use to verify this master is in sync with it.
Raises:

spicerack.mysql.MysqlError -- on failure to gather the heartbeat or convert it into a datetime or not yet in sync.
static _get_heartbeat(mysql_hosts, section, heartbeat_dc)[source]

Get the heartbeat from the remote host for a given DC.

Parameters:
  • mysql_hosts (spicerack.mysql.MysqlRemoteHosts) -- the instance for the target DB to query.
  • section (str) -- the DB section for which to get the heartbeat.
  • heartbeat_dc (str) -- the name of the datacenter for which to filter the heartbeat query.
Returns:

the converted heartbeat.
Return type:

datetime.datetime
Raises:

spicerack.mysql.MysqlError -- on failure to gather the heartbeat or convert it into a datetime.
check_core_masters_heartbeats(datacenter, heartbeat_dc, heartbeats)[source]

Check the current heartbeat values in the core DB masters in DC are in sync with the provided heartbeats.

Parameters:
  • datacenter (str) -- the name of the datacenter from where to get the heartbeat values.
  • heartbeat_dc (str) -- the name of the datacenter for which to filter the heartbeat query.
  • heartbeats (dict) -- a dictionary with section (str): heartbeat (datetime.datetime) for each core section.
Raises:

spicerack.mysql.MysqlError -- on failure to gather the heartbeat or convert it into a datetime.
check_core_masters_in_sync(dc_from, dc_to)[source]

Check that all core masters in dc_to are in sync with the core masters in dc_from.

Parameters:
  • dc_from (str) -- the name of the datacenter from where to get the master positions.
  • dc_to (str) -- the name of the datacenter where to check that they are in sync.
Raises:

spicerack.remote.RemoteExecutionError -- on failure.
get_core_dbs(*, datacenter=None, section=None, replication_role=None)[source]

Find the core databases matching the parameters.

Parameters:
  • datacenter (str, optional) -- the name of the datacenter to filter for, accepted values are those specified in spicerack.CORE_DATACENTERS.
  • replication_role (str, optional) -- the repication role to filter for, accepted values are those specified in spicerack.mysql.REPLICATION_ROLES.
  • section (str, optional) -- a specific section to filter for, accepted values are those specified in spicerack.mysql.CORE_SECTIONS.
Raises:

spicerack.mysql.MysqlError -- on invalid data or unexpected matching hosts.
Returns:

an instance with the remote targets.
Return type:

spicerack.mysql.MysqlRemoteHosts
get_core_masters_heartbeats(datacenter, heartbeat_dc)[source]

Get the current heartbeat values from core DB masters in DC for a given heartbeat DC.

Parameters:
  • datacenter (str) -- the name of the datacenter from where to get the heartbeat values.
  • heartbeat_dc (str) -- the name of the datacenter for which to filter the heartbeat query.
Returns:

a dictionary with section (str): heartbeat (datetime.datetime) for each core section. For example:

{'s1': datetime.datetime(2018, 1, 2, 11, 22, 33, 123456)}
Return type:

dict
Raises:

spicerack.mysql.MysqlError -- on failure to gather the heartbeat or convert it into a datetime.
get_dbs(query)[source]

Get a MysqlRemoteHosts instance for the matching hosts.

Parameters:query (str) -- the Remote query to use to fetch the DB hosts.
Returns:an instance with the remote targets.
Return type:spicerack.mysql.MysqlRemoteHosts
heartbeat_query = "SELECT ts FROM heartbeat.heartbeat WHERE datacenter = '{dc}' and shard = '{section}' ORDER BY ts DESC LIMIT 1"

Query pattern to check the heartbeat for a given datacenter and section.

set_core_masters_readonly(datacenter)[source]

Set the core masters in read-only.

Parameters:

datacenter (str) -- the name of the datacenter to filter for.
Raises:
set_core_masters_readwrite(datacenter)[source]

Set the core masters in read-write.

Parameters:

datacenter (str) -- the name of the datacenter to filter for.
Raises:
verify_core_masters_readonly(datacenter, is_read_only)[source]

Verify that the core masters are in read-only or read-write mode.

Parameters:
  • datacenter (str) -- the name of the datacenter to filter for.
  • is_read_only (bool) -- whether the read-only mode should be set or not.
Raises:

spicerack.mysql.MysqlError -- on failure.
exception spicerack.mysql.MysqlError[source]

Bases: spicerack.exceptions.SpicerackError

Custom exception class for errors of this module.

class spicerack.mysql.MysqlRemoteHosts(remote_hosts)[source]

Bases: spicerack.remote.RemoteHostsAdapter

Custom RemoteHosts class to execute MySQL queries.

__init__(remote_hosts)

static inherited Initialize the instance.

Parameters:remote_hosts (spicerack.remote.RemoteHosts) -- the instance to act on the remote hosts.
run_query(query, database='', success_threshold=1.0, batch_size=None, batch_sleep=None, is_safe=False)[source]

Execute the query via Remote.

Parameters:
  • query (str) -- the mysql query to be executed. Double quotes must be already escaped.
  • database (str, optional) -- an optional MySQL database to connect to before executing the query.
  • success_threshold (float, optional) -- to consider the execution successful, must be between 0.0 and 1.0.
  • batch_size (int, str, optional) -- the batch size for cumin, either as percentage (i.e. '25%') or absolute number (i.e. 5).
  • batch_sleep (float, optional) -- the batch sleep in seconds to use in Cumin before scheduling the next host.
  • is_safe (bool, optional) -- whether the command is safe to run also in dry-run mode because it's a read-only command that doesn't modify the state.
Returns:

cumin.transports.BaseWorker.get_results to allow to iterate over the results.
Return type:

generator
Raises:

RemoteExecutionError -- if the Cumin execution returns a non-zero exit code.
spicerack.mysql.REPLICATION_ROLES = ('master', 'slave', 'standalone')

list of valid replication roles.

Type:tuple