Python API

Spicerack Python API autodoc.

Spicerack package.

class spicerack.Spicerack(*, verbose: bool = False, dry_run: bool = True, cumin_config: str = '/etc/cumin/config.yaml', cumin_installer_config: str = '/etc/cumin/config-installer.yaml', conftool_config: str = '/etc/conftool/config.yaml', conftool_schema: str = '/etc/conftool/schema.yaml', debmonitor_config: str = '/etc/debmonitor.conf', etcd_config: str = '', spicerack_config_dir: str = '/etc/spicerack', http_proxy: str = '', get_cookbook_callback: collections.abc.Callable[[Spicerack, str, collections.abc.Sequence[str]], BaseItem | None] | None = None, extender_class: type[SpicerackExtenderBase] | None = None) → None[source]

Bases: object

Spicerack service locator.

Initialize the service locator for the Spicerack library.

Parameters:

  • verbose (bool, default: False) -- whether to set the verbose mode.

  • dry_run (bool, default: True) -- whether this is a DRY-RUN.

  • cumin_config (str, default: '/etc/cumin/config.yaml') -- the path to Cumin's configuration file.

  • cumin_installer_config (str, default: '/etc/cumin/config-installer.yaml') -- the path to Cumin's configuration file for the Debian installer.

  • conftool_config (str, default: '/etc/conftool/config.yaml') -- the path to Conftool's configuration file.

  • conftool_schema (str, default: '/etc/conftool/schema.yaml') -- the path to Conftool's schema file.

  • debmonitor_config (str, default: '/etc/debmonitor.conf') --

    the path to Debmonitor's INI configuration file. It must have at least the following schema:

    [DEFAULT]
server=debmonitor.example.com
cert=/etc/debmonitor/ssl/cert.pem
key=/etc/debmonitor/ssl/server.key

  • etcd_config (str, default: '') -- the path to the Etcd configuration file. Set to empty string to disable the locking support.

  • spicerack_config_dir (str, default: '/etc/spicerack') -- the path for the root configuration directory for Spicerack. Module-specific configuration will be loaded from config_dir/module_name/.

  • http_proxy (str, default: '') -- the scheme://url:port of the HTTP proxy to use for external calls.

  • get_cookbook_callback (typing.Optional[collections.abc.Callable[[Spicerack, str, collections.abc.Sequence[str]], typing.Optional[spicerack._menu.BaseItem]]], default: None) -- a callable to retrieve a CookbookItem to execute a cookbook from inside another cookbook.

  • extender_class (typing.Optional[type[SpicerackExtenderBase]], default: None) -- an optional class object that inherits from spicerack.SpicerackExtenderBase to dynamically add accessors to Spicerack. If not set no extenders will be registered, even if external_modules_dir is specified in the configuration.

__getattr__(name: str) → Any[source]

Attribute accessor to dynamically load external accessors when present.

This method is called only if a method or attribute with the given name is not present in the current instance.

Parameters:

according to Python's Data model object.__getattr__().

Return type:

typing.Any

admin_reason(reason: str, task_id: str | None = None) → spicerack.administrative.Reason[source]

Get an administrative Reason instance.

Parameters:
Return type:

spicerack.administrative.Reason

alerting_hosts(target_hosts: spicerack.typing.TypeHosts, *, verbatim_hosts: bool = False) → spicerack.alerting.AlertingHosts[source]

Get an AlertingHosts instance.

Parameters:
Return type:

spicerack.alerting.AlertingHosts

alertmanager(instance_name: str = '') → spicerack.alertmanager.Alertmanager[source]

Get an Alertmanager instance.

Parameters:

instance_name (str, default: '') -- the Alertmanager instance defined in the alertmanager/config.yaml config file to interact with

Return type:

spicerack.alertmanager.Alertmanager

Note

To interact with Alertmanager alerts attached to an instance use spicerack.Spicerack.alertmanager_hosts() instead.

alertmanager_hosts(target_hosts: spicerack.typing.TypeHosts, *, instance_name: str = '', verbatim_hosts: bool = False) → spicerack.alertmanager.AlertmanagerHosts[source]

Get an AlertmanagerHosts instance.

Note

To interact with both Icinga and Alertmanager alerts, use spicerack.Spicerack.alerting_hosts() instead.

Parameters:

  • target_hosts (typing.TypeVar(TypeHosts, collections.abc.Sequence[str], ClusterShell.NodeSet.NodeSet)) -- the target hosts either as a NodeSet instance or a sequence of strings.

  • instance_name (str, default: '') -- the Alertmanager instance defined in the alertmanager/config.yaml config file to interact with

  • verbatim_hosts (bool, default: False) -- if True use the hosts passed verbatim as is, if instead False, the default, consider the given target hosts as FQDNs and extract their hostnames to be used in Icinga.

Return type:

spicerack.alertmanager.AlertmanagerHosts

api_client(base_url: str, accept_header: str = 'application/json', **kwargs: Any) → spicerack.apiclient.APIClient[source]

Return a generic APIClient instance with the given base URL and HTTP session based on the parameters.

Parameters:

  • base_url (str) -- the full base URL for the API. It must include the scheme and the domain and can include the API path prefix if there is one.

  • accept_header (str, default: 'application/json') -- an optional HTTP Accept header to set in the HTTP session. By default a JSON API is assumed.

  • **kwargs (typing.Any) -- arbitrary keyword arguments passed directly to wmflib.requests.http_session() to customize the HTTP session used for this API.

Return type:

spicerack.apiclient.APIClient

Returns:

A generic API client instance with DRY-RUN support.

apt_get(remote_hosts: spicerack.remote.RemoteHosts) → spicerack.apt.AptGetHosts[source]

Get an APTGet instance for the given remote hosts.

Examples

>>> hosts = spicerack.remote().query('A:myalias')
>>> apt_get = spicerack.apt_get(hosts)
Parameters:

remote_hosts (spicerack.remote.RemoteHosts) -- the instance with the target hosts.

Return type:

spicerack.apt.AptGetHosts

confctl(entity_name: str) → spicerack.confctl.ConftoolEntity[source]

Get a Conftool specific entity instance.

Parameters:

entity_name (str) -- the name of a Conftool entity. Possible values: node, service, discovery, mwconfig.

Return type:

spicerack.confctl.ConftoolEntity

dbctl() → spicerack.dbctl.Dbctl[source]

Get a Dbctl instance to interact with dbctl.

Return type:

spicerack.dbctl.Dbctl

debmonitor() → spicerack.debmonitor.Debmonitor[source]

Get a Debmonitor instance to interact with a Debmonitor website.

Raises:

KeyError -- if any configuration option is missing.

Return type:

spicerack.debmonitor.Debmonitor

dhcp(datacenter: str) → spicerack.dhcp.DHCP[source]

Return a DHCP configuration manager for the specified datacenter.

Parameters:

datacenter (str) -- the datacenter for which the DHCP servers will be targeted.

Return type:

spicerack.dhcp.DHCP

discovery(*records: str) → spicerack.dnsdisc.Discovery[source]

Get a Discovery instance for the given records.

Parameters:

*records (str) -- arbitrary positional arguments, each one must be a Discovery DNS record name.

Return type:

spicerack.dnsdisc.Discovery

dns() → wmflib.dns.Dns[source]

Get a Dns instance that will use the operating system default nameserver(s).

Return type:

wmflib.dns.Dns

elasticsearch_clusters(clustergroup: str, write_queue_datacenters: collections.abc.Sequence[str]) → spicerack.elasticsearch_cluster.ElasticsearchClusters[source]

Get an ElasticsearchClusters instance.

Parameters:

  • clustergroup (str) -- name of cluster group e.g search_eqiad.

  • write_queue_datacenters (collections.abc.Sequence[str]) -- Sequence of which core DCs to query write queues for.

Return type:

spicerack.elasticsearch_cluster.ElasticsearchClusters

etcdctl(*, remote_host: spicerack.remote.RemoteHosts) → spicerack.toolforge.etcdctl.EtcdctlController[source]

Add etcdctl control capabilities to the given RemoteHost.

Params:
remote_host: Single remote host (that should map one single host)

to add capabilities to.

Return type:

spicerack.toolforge.etcdctl.EtcdctlController

Returns:

A wrapped RemoteHost with the etcdctl control related methods.

ganeti() → spicerack.ganeti.Ganeti[source]

Get an instance to interact with Ganeti.

Raises:

KeyError -- If the configuration file does not contain the correct keys.

Return type:

spicerack.ganeti.Ganeti

host(name: str, *, netbox_read_write: bool = False) → spicerack.hosts.Host[source]

Get a Host instance that represents a single physical server or virtual machine.

It exposes a series of accessors like Spicerack but tailored for a single host.

Parameters:

  • name (str) -- the short hostname of the host as present in Netbox.

  • netbox_read_write (bool, default: False) -- if the Netbox token used should allow read-write operations or not.

Return type:

spicerack.hosts.Host

Returns:

the Host instance.

Raises:

spicerack.hosts.HostError -- if unable to instantiate the Host instance.

icinga_hosts(target_hosts: spicerack.typing.TypeHosts, *, verbatim_hosts: bool = False) → spicerack.icinga.IcingaHosts[source]

Get an IcingaHosts instance.

Note

To interact with both Icinga and Alertmanager alerts, use spicerack.Spicerack.alerting_hosts() instead.

Parameters:
Return type:

spicerack.icinga.IcingaHosts

icinga_master_host() → spicerack.remote.RemoteHosts[source]

Returns the instance to execute commands on the Icinga master host.

Return type:

spicerack.remote.RemoteHosts

ipmi(target: str) → spicerack.ipmi.Ipmi[source]

Get an Ipmi instance to send remote IPMI commands to management consoles.

Parameters:

target (str) -- the management console FQDN or IP to target.

Return type:

spicerack.ipmi.Ipmi

kafka() → spicerack.kafka.Kafka[source]

Get an instance to interact with Kafka.

Raises:

KeyError -- If the configuration file does not contain the correct keys.

Return type:

spicerack.kafka.Kafka

kubernetes(group: str, cluster: str) → spicerack.k8s.Kubernetes[source]

Get a kubernetes client for the specified cluster.

Parameters:

  • group (str) -- the cluster group.

  • cluster (str) -- the kubernetes cluster.

Return type:

spicerack.k8s.Kubernetes

lock() → spicerack.locking.Lock | spicerack.locking.NoLock[source]

Get a Lock instance to acquire custom locks with concurrency and TTL around specific lines of code.

In case the locking support is disabled, a spicerack.locking.NoLock instance is returned, with the same API of the spicerack.locking.Lock class but that does nothing.

Return type:

typing.Union[spicerack.locking.Lock, spicerack.locking.NoLock]

Returns:

The lock instance already initialized with a dedicated prefix for the keys.

management_password() → str[source]

Returns the management password.

It will be asked to the user if not already cached by the current instance.

Return type:

str

mediawiki() → spicerack.mediawiki.MediaWiki[source]

Get a MediaWiki instance.

Return type:

spicerack.mediawiki.MediaWiki

mysql() → spicerack.mysql.Mysql[source]

Get a Mysql instance.

Return type:

spicerack.mysql.Mysql

netbox(*, read_write: bool = False) → spicerack.netbox.Netbox[source]

Get a Netbox instance to interact with Netbox's API.

Parameters:

read_write (bool, default: False) -- whether to use a read-write token.

Return type:

spicerack.netbox.Netbox

netbox_master_host() → spicerack.remote.RemoteHosts[source]

Returns the instance to execute commands on the Netbox master host.

Return type:

spicerack.remote.RemoteHosts

netbox_server(hostname: str, *, read_write: bool = False) → spicerack.netbox.NetboxServer[source]

Get a NetboxServer instance to interact with a server in Netbox, both physical and virtual.

Parameters:

  • hostname (str) -- the hostname (not FQDN) of the server to manage.

  • read_write (bool, default: False) -- whether to use a read-write token.

Raises:

spicerack.netbox.NetboxError -- if unable to find or load the server.

Return type:

spicerack.netbox.NetboxServer

orchestrator() → spicerack.orchestrator.Orchestrator[source]

Get an instance to interact with the Orchestrator APIs.

Return type:

spicerack.orchestrator.Orchestrator

Returns:

the orcestrator instance.

peeringdb(*, ttl: int = 86400) → spicerack.peeringdb.PeeringDB[source]

Get a PeeringDB instance to interact with the PeeringDB API.

Parameters:

ttl (int, default: 86400) -- the cache timeout in seconds. If cached items are older than the given TTL they will be ignored and fetched again.

Return type:

spicerack.peeringdb.PeeringDB

phabricator(bot_config_file: str, section: str = 'phabricator_bot') → wmflib.phabricator.Phabricator[source]

Get a Phabricator instance to interact with a Phabricator website.

The Phabricator object is instantiated with allow_empty_identifiers=True, so that it can be a NOOP when executed with an empty string task ID or other identifier.

Parameters:

  • bot_config_file (str) --

    the path to the configuration file for the Phabricator bot, with the following structure:

    [section_name]
host = https://phabricator.example.com/api/
username = phab-bot
token = api-12345

  • section (str, default: 'phabricator_bot') -- the name of the section of the configuration file where to find the required parameters.

Return type:

wmflib.phabricator.Phabricator

prometheus() → wmflib.prometheus.Prometheus[source]

Get a Prometheus instance.

Return type:

wmflib.prometheus.Prometheus

puppet(remote_hosts: spicerack.remote.RemoteHosts) → spicerack.puppet.PuppetHosts[source]

Get a PuppetHosts instance for the given remote hosts.

Parameters:

remote_hosts (spicerack.remote.RemoteHosts) -- the instance with the target hosts.

Return type:

spicerack.puppet.PuppetHosts

puppet_master() → spicerack.puppet.PuppetMaster[source]

Get a PuppetMaster instance to manage hosts and certificates from a Puppet master.

Return type:

spicerack.puppet.PuppetMaster

puppet_server() → spicerack.puppet.PuppetServer[source]

Get a PuppetServer instance to manage hosts and certificates from a Puppet master.

Return type:

spicerack.puppet.PuppetServer

redfish(hostname: str, username: str = 'root', password: str = '') → spicerack.redfish.Redfish[source]

Get an instance to talk to the Redfish API of a physical server.

Notes

At the moment only Dell hardware is supported.

Parameters:

  • hostname (str) -- the hostname (not FQDN) of the physical server to manage.

  • username (str, default: 'root') -- the username for the management console.

  • password (str, default: '') -- the password for the management console for the given user. If empty or not provided would use the production management password and ask the user for it if not already in memory.

Raises:

spicerack.exceptions.SpicerackError -- if not a physical server or unable to find the management IP.

Return type:

spicerack.redfish.Redfish

redis_cluster(cluster: str) → spicerack.redis_cluster.RedisCluster[source]

Get a RedisCluster instance.

Parameters:

cluster (str) -- the name of the cluster.

Return type:

spicerack.redis_cluster.RedisCluster

remote(installer: bool = False) → spicerack.remote.Remote[source]

Get a Remote instance.

Parameters:

installer (bool, default: False) -- whether to use the special configuration to connect to a Debian installer or freshly re-imaged host prior to its first Puppet run.

Return type:

spicerack.remote.Remote

reposync(name: str) → spicerack.reposync.RepoSync[source]

Get a Reposync instance.

Parameters:

name (str) -- the name of the repo to sync.

Return type:

spicerack.reposync.RepoSync

requests_session(name: str, **kwargs: Any) → requests.sessions.Session[source]

Return a new requests Session with timeout and retry logic.

Return type:

requests.sessions.Session

Params:

according to wmflib.requests.http_session().

run_cookbook(cookbook: str, args: collections.abc.Sequence[str] = (), *, raises: bool = False, confirm: bool = False) → int[source]

Run another Cookbook within the current run.

The other Cookbook will be executed with the current setup and will log in the same file of the current Cookbook that is running. This includes the propagation of DRY-RUN mode to the called cookbooks.

Parameters:

  • cookbook (str) -- the path to the cookbook to execute, either in Spicerack's dot notation or the relative path to the Python file to execute.

  • args (collections.abc.Sequence[str], default: ()) -- an iterable sequence of strings with the Cookbook's argument. The Cookbook will be executed with the same global arguments used for the current run.

  • raises (bool, default: False) -- if set to True raises a spicerack.exceptions.RunCookbookError exception in case the cookbook execution returns a non-zero exit code.

  • confirm (bool, default: False) -- if set to True wraps the call of the cookbook in a wmflib.interactive.confirm_on_failure() call. It automatically sets raises to True.

Return type:

int

Returns:

The exit code of the Cookbook, 0 if successful, non-zero if not, unless raises is set to True, in that case it will raise instead.

Raises:
service_catalog(*, refresh: bool = False) → spicerack.service.Catalog[source]

Get a Catalog instance that reflects Puppet's service::catalog hieradata variable.

The catalog is cached until explicitly refreshed.

Parameters:

refresh (bool, default: False) -- when set to True forces a refresh of the catalog data from disk and caches the new one.

Return type:

spicerack.service.Catalog

Returns:

The service catalog data and caches it.

thanos() → wmflib.prometheus.Thanos[source]

Get a Thanos instance.

Return type:

wmflib.prometheus.Thanos

property actions: ActionsDict

Returns a dictionary to log and record cookbook actions.

property authdns_active_hosts: RemoteHosts

Get a RemoteHosts instance to target the active authoritative nameservers.

Examples

>>> authdns_hosts = spicerack.authdns_active_hosts
>>> authdns_hosts.run_sync('some command')
property authdns_servers: dict[str, str]

Getter for the authoritative DNS nameservers currently active in production.

Returns a dictionary where keys are the hostnames and values are the IPs of the active authoritative nameservers.

property config_dir: Path

Getter for Spicerack's configuration file directory where the module-specific configs are available.

They can be accessed with:

self.config_dir / module_name / config_file
property current_hostname: str

Returns the name of the hostname where the code is running.

property dry_run: bool

Returns True if the DRY-RUN mode is set, False otherwise.

property http_proxy: str

Returns the scheme://url:port of the HTTP PROXY proxy.

property irc_logger: Logger

Returns the logger instance to write to IRC in the #wikimedia-operations channel.

property owner: str

user@host [pid].

Type:

Returns the owner of the current code

property requests_proxies: dict[str, str] | None

Returns the HTTP proxy configuration for the Requests module or None if no HTTP proxy is set.

See also

https://requests.readthedocs.io/en/master/user/advanced/#proxies

property sal_logger: Logger

Returns the logger instance to write to IRC in the #wikimedia-operations and logging to SAL.

property username: str

Returns the name of the effective running user.

property verbose: bool

Returns True if the verbose mode is set, False otherwise.

class spicerack.SpicerackExtenderBase(*, spicerack: spicerack.Spicerack)[source]

Bases: object

Base class to create a Spicerack extender. Necessary when the external_modules_dir configuration is set.

Initialize the instance.

Parameters:

spicerack (spicerack.Spicerack) -- the Spicerack instance.

spicerack.__version__: str = '0.1.dev2+g1aa505ccd'

The version of the current Spicerack module.

Subpackages and Submodules