Python API¶
Core Application Infrastructure¶
scap¶
Wikimedia’s MediaWiki deployment script. Deploys MediaWiki code and configuration to a group of servers via SSH and rsync.
Copyright © 2014-2017 Wikimedia Foundation and Contributors.
This file is part of Scap.
Scap is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, version 3.
This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
You should have received a copy of the GNU General Public License along with this program. If not, see <http://www.gnu.org/licenses/>.
scap.cli¶
Classes and helpers for creating command line interfaces
Copyright © 2014-2017 Wikimedia Foundation and Contributors.
This file is part of Scap.
Scap is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, version 3.
This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
You should have received a copy of the GNU General Public License along with this program. If not, see <http://www.gnu.org/licenses/>.
- class scap.cli.Application(exe_name)[source]¶
Base class for creating command line applications.
- _before_exit(exit_status)[source]¶
Do any final cleanup or processing before the application exits.
Called after
main()
and before sys.exit even when an exception occurs.- Returns:
exit status
- _check_user_auth_sock()[source]¶
Similar to _assert_auth_sock method above.
Returns immediately when “gerrit_push_user” has been configured, it is assumed the identity is provided in the ssh authentication socket / keyholder.
- Unlike _assert_auth_sock, this method:
Checks the original SSH_AUTH_SOCK provided in the environment by the user, before scap potentially overrides it
Is meant to be called during operations where a user-provided ssh-agent is required (e.g. deploy-promote). It is not a global requirement of scap
- _process_arguments(args, extra_args)[source]¶
Validate and process command line arguments.
Default behavior is to abort the application with an error if any unparsed arguments were found.
- Returns:
Tuple of (args, extra_args) after processing
- active_wikiversions(source_tree='deploy', return_type=<class 'list'>)[source]¶
Get an ordered list or dictionary of active MediaWiki versions.
- Parameters:
source_tree – Source tree to read file from: ‘deploy’ or ‘stage’
return_type – One of list or dict.
- Returns:
If return_type is list (the default), returns a list of version strings (like “1.38.0-wmf.4”) in ascending order.
If return_type is dict, returns a collections.OrderedDict of {version:wikidb} values sorted by version number in ascending order. ‘wikidb’ will be the alphabetically-first wikidb for ‘version’. This can be used by operations that need a db but don’t care which wiki’s db is used.
- announce(*args)[source]¶
Announce a message to broadcast listeners.
Emits a logging event to the ‘scap.announce’ logger which can be configured to broadcast messages via irc or another real-time notification channel.
Announcements can be disabled by using ‘–no-log-message’ in the command invocation (e.g. scap sync-file –no-log-message foo.php). In this case the log event will still be emitted to the normal logger as though self.get_logger().info() was used instead of self.announce().
- announce_final(*args)[source]¶
Use this method to avoid announcing (to IRC, etc) that an operation has finished if there was no prior announcement that an operation had started. This is useful is situations where self.announce() is called in an except or finally clause surrounding a larger operation.
- format_passthrough_args() list [source]¶
Returns a list with user-supplied cli config args in cli format. Makes it easy to pass along config args to locally spawned scap processes.
For example, if the user passed config options canary_dashboard_url and log_json on the command line, then the return value would be:
[“-D”, “canary_dashboard_url:https://somewhere.over.the.rainbow”, “-D”, “log_json:false”]
- get_gerrit_ssh_env() dict [source]¶
An environment to interact with Gerrit over ssh
When gerrit_push_user scap configuration is set, set the GIT_SSH_COMMAND environment variable to have git ssh to use that user instead of the username of the person that invoked scap.
Else if a user-supplied ssh agent socket was provided, override the one set by scap which does not have the invoking user key.
Eventually we will always require gerrit_push_user to be set.
See https://phabricator.wikimedia.org/T304557
- Returns:
environment suitable for sshing to Gerrit
- get_keyholder_key(ssh_user=None, key_name=None)[source]¶
Get the private key for IdentityFile use in ssh.
- get_master_list(limit_hosts=None, exclude_hosts=None)[source]¶
Get list of deploy master hostnames that should be updated before the rest of the cluster.
- get_mediawiki_staging_lock_file()[source]¶
Get the path to the lock file corresponding to the MediaWiki staging directory
- lock_mediawiki_staging(**kwargs)[source]¶
Acquire a lock for the main work of this MediaWiki staging related application.
- lock_mediawiki_staging_and_announce(**kwargs)[source]¶
Acquire a lock for the main work of this application and announce its start and finish.
- main(*extra_args)[source]¶
Main business logic of the application.
Parsed command line arguments are available in self.arguments. Global configuration is available in self.config. Unparsed command line arguments are passed as positional arguments.
- Returns:
exit status
- property message_argument: str¶
The given message
argument()
if one was defined, otherwise it returns “(no justification provided)”.
- prompt_for_approval_or_exit(prompt_message, exit_message)[source]¶
Exits successfully with a message if the user does not approve.
- reported_status(status: str, log=False)[source]¶
Set status to status, do work, then clear the job status.
- classmethod run()[source]¶
Construct and run an application.
Calls
sys.exit
with exit status returned by application by default. Settingexit
toFalse
will instead return the class instance and it’s exit status. This would generally only be done when testing.- Parameters:
cls – Class to create and run
argv – Command line arguments
- Returns:
Tuple of class instance and exit status when not exiting
- scap_call(scap_cmd: list, user=None, env=None, passthrough_arguments=True, logger=None, **kwargs) CompletedProcess [source]¶
Call scap subcommand
- Returns:
the completed process
- scap_check_call(scap_cmd: list, **kwargs) CompletedProcess [source]¶
Call scap subcommand and check for a non-zero exit status
- Returns:
the completed process
- Raises:
CalledProcessError – if the subcommand fails
- @scap.cli.argument(option_flags..[,action='store'][,nargs=1] [,const=None][,default][,type=str][,choices][,required=False][,help] [,dest])[source]¶
Decorator used to declare a command line argument on an
Application
Maps a command line argument to the decorated class or method.
- Parameters:
option_flags (str) – One or more option flags associated with this argument, e.g. ‘-a’, ‘–arg’
action – the action associated with this argument. e.g. ‘store_true’
default – The default value for this argument if not specified by the user.
help (str) – Short description of this argument, displayed in
--help
text.choices (list) – List possible values for this argument. If specified, argument values will be validated for you and
--help
will list the possible choices for the user.required (bool) – True if your argument is required.
type (type) – The type of value accepted by your argument, e.g. int or file.
nargs (int, str) – The number of values accepted by this argument.
- @scap.cli.command(command_name, help="help text"[, subcommands=False])[source]¶
Map a scap sub-command to the decorated class.
- Parameters:
- Raises:
ValueError – if there is already a command named command_name
Usage Example:
import scap.cli @cli.command('hello', help='prints "hello world" and exits') class HelloCommand(cli.Application): @cli.argument('--goodbye', action='store_true', help='Say goodbye instead.') def main(extra_args): if self.arguments.goodbye: print('Goodbye, cruel world.') else: print('Hello, world.')
- @scap.cli.subcommand(command_name)[source]¶
Define an argparse subcommand by decorating a method on your cli.Application subclass.
Use this decorator when you want to have a subcommand on a method other than ‘main’
In order for this to have any affect, your cli.Application must be decorated with subcommands=True (see example below).
Usage Example:
import scap.cli @cli.command('hello', subcommands=True, help='prints "hello world" and exits',) class HelloCommand(cli.Application): @cli.subcommand('world') def world_subcommand(extra_args): print('hello world')
scap.arg¶
Helpers for creating a fancy argparser. Most of the externally useful API for command line arg parsing is found in scap.cli
- Author:
Tyler Cipriani <thcipriani@wikimedia.org>
- Author:
Mukunda Modell <mmodell@wikimedia.org>
- Copyright:
Wikimedia Foundation, Inc.
- License:
GPL v3.0
Parts of the shell argument completion code in this file is derived from python-selfcompletion.
- Author:
David Barnett <davidbarnett2@gmail.com>
- License:
BSD
See also
Copyright © 2014-2017 Wikimedia Foundation and Contributors.
This file is part of Scap.
Scap is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, version 3.
This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
You should have received a copy of the GNU General Public License along with this program. If not, see <http://www.gnu.org/licenses/>.
- class scap.arg.ScapArgParser(*args, **kwargs)[source]¶
Scap argparse subclass
Created to allow for easier, scripted, autocompletion
- class scap.arg.ScapHelpFormatter(prog, indent_increment=2, max_help_position=32, width=None)[source]¶
Formatter that respects argparse.SUPPRESS for subparser actions.
- scap.arg.build_subparser(cmd, parser, global_parser)[source]¶
Append subparsers to
cli.Application
’s argparser using decorators.
- scap.arg.get_global_parser()[source]¶
Add standard arguments to argparser.
These arguments should be present on all subparsers.
- ..info::
The other option with these commands would be to make them into top-level flags for scap; however, that ends up feeling clunky with commands like:
scap –verbose sync
Or
scap -e beta deploy –repo mockbase/deploy
scap.config¶
Configuration management
Copyright © 2014-2017 Wikimedia Foundation and Contributors.
This file is part of Scap.
Scap is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, version 3.
This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
You should have received a copy of the GNU General Public License along with this program. If not, see <http://www.gnu.org/licenses/>.
- scap.config.coerce_value(key, value)[source]
Coerce the given value based on the default config type.
- scap.config.load(cfg_file=None, environment=None, overrides=None, use_global_config=True, use_local_config=True)[source]
Load configuration.
A configuration file consists of sections, led by a
[section]
header and followed byname: value
entries. Lines beginning with'#'
are ignored and may be used to provide comments.A configuration file can contain multiple sections. The configuration object is populated with values from the
global
section and additional sections based on the fully qualified domain name of the local host. For example, on the hostdeployXXXX.eqiad.wmnet
the final value for a given setting would be the first value found in sections:deployXXXX.eqiad.wmnet
,eqiad.wmnet
,wmnet
orglobal
. Sections not present in the configuration file will be ignored.Configuration values are loaded from a file specified by the
-c
or--conf
command-line options or from the default locations with the following hierarchy, sorted by override priority:$(pwd)/scap/environments/<environment>/scap.cfg
or$(pwd)/scap/scap.cfg
(if no environment was specified)/etc/scap.cfg
(if use_global_config is true)
For example, if a configuration parameter is set in
$(pwd)/scap/scap.cfg
and that same parameter is set in/etc/scap.cfg
the value for that parameter set in$(pwd)/scap/scap.cfg
will be used during execution.- Parameters:
cfg_file – Alternate configuration file
environment – the string path under which scap.cfg is found
overrides – Dict of configuration values
use_global_config – A boolean indicating if /etc/scap.cfg should be read
- Returns:
dict of configuration values
- scap.config.multi_value(str_value)[source]
Given a string that’s got commas, turn it into a list
- Parameters:
str_value – Random thing the user typed in config
- scap.config.override_config(config, overrides=None)[source]
Override values in a config with type-coerced values.
Main scap command classes¶
scap.main¶
Command wrappers for scap tasks
Copyright © 2014-2017 Wikimedia Foundation and Contributors.
This file is part of Scap.
Scap is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, version 3.
This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
You should have received a copy of the GNU General Public License along with this program. If not, see <http://www.gnu.org/licenses/>.
- class scap.main.AbstractSync(exe_name)[source]¶
Base class for applications that want to sync one or more files from the deployment server to the rest of the cluster.
- _apache_sync_command(proxies: list = (), **kwargs) list [source]¶
Returns (as a list) the scap pull command to run on mediawiki installation targets. This is comprised of the base scap pull command (defined by _base_scap_pull_command) followed by the list of deployment masters and the list of proxies.
- Parameters:
proxies – A list of proxy hostnames that can be pulled from in addition to the deployment masters. Default is empty (as a tuple to prevent mutable list warning)
kwargs – Any remaining keyword arguments are passed on to _base_scap_pull_command.
- _base_scap_pull_command() list [source]¶
Returns (as a list) the basic scap pull command to run on a remote target. Note that no source servers are specified in the command so scap pull will default to pull from whatever master_rsync is defined to be in the scap configuration on the target.
- _get_proxy_list()[source]¶
Get list of sync proxy hostnames that should be updated before the rest of the cluster.
- _perform_sync(type: str, command: list, targets: list, shuffle=False)[source]¶
- Parameters:
type – A string like “apaches” or “proxies” naming the type of target.
command – The command to run on the targets, specified as a list.
targets – A list of strings naming hosts to sync.
shuffle – If true, the target host list will be randomized.
- _restart_php()[source]¶
If php_fpm_restart_script is set in the configuration then on all dsh groups referenced by the mw_web_clusters config parameter do the following:
Check if php-fpm opcache is full, if so restart php-fpm. If the php_fpm_always_restart config parameter is true, the opcache is treated as always full, so php-fpm will always restart.
If the operator invoked scap with the –force flag, restart php-fpm unsafely (i.e., without depooling and repooling around the service restart). T243009
Targets that have already been restarted will not be restarted again.
- _restart_php_hostgroups(target_hosts=None)[source]¶
Perform php restart for sets of hosts (if configured).
Parameter target_hosts is a list of lists of hostnames.
- _setup_php()[source]¶
Sets up the php_fpm instance if not already initialized.
Returns True if php_fpm restart has been configured, or False if not.
- canary_checks(baremetal_canaries: list)[source]¶
Run logstash error rate check (for bare metal and mw-on-k8s).
- Raises:
SystemExit – on canary check failure
- check_testservers(baremetal_testservers: list)[source]¶
Check bare metal and k8s testservers.
- Raises:
SystemExit – on check failure
- get_keyholder_key(*args, **kwargs)[source]¶
Returns scap2-specific deploy key
This way we can set a key in the default scap config without having all non-scap2 repos inherit that configuration.
- increment_stat(stat, all_stat=True, value=1)[source]¶
Increment a stat in deploy.*
- Parameters:
stat – String name of stat to increment
all_stat – Whether to increment deploy.all as well
value – How many to increment by, default of 1 is normal
- master_only_cmd(timer, cmd)[source]¶
Run a command on all other master servers than the one we’re on
- Parameters:
timer – String name to use in timer/logging
cmd – List of command/parameters to be executed
- retry_continue_exit(description, test_func)[source]¶
Runs test_func(). If it returns True, this function returns. If test_func() does not return true:
If an interactive terminal is not available, self.cancel() is called and it is not expected to return.
If an interactive terminal is available, ask the user if they want to retry the test, continue with deployment anyway, or exit. If the user chooses to retry, this function starts over. If the user chooses to continue, a message is logged and this function returns. If the user chooses to exit, self.cancel() is called and it is not expected to return.
‘description’ is used to describe the operation to retry in the retry/continue/exit prompt.
- sync_targets(targets=None, type=None)[source]¶
This function is used to sync to bare metal testservers and canaries.
Run scap pull on the targets, including l10n rebuild, wikiversions sync, and php-fpm restart. The pull source will be this deploy server.
- Parameters:
targets – Iterable of target servers to sync
type – A string like “testservers” or “canaries” naming the type of target.
- class scap.main.CompileWikiversions(exe_name)[source]¶
Compile wikiversions.json to wikiversions.php.
- class scap.main.DeploymentStage(name: str, baremetal_targets: List[str], check_func: Union[Callable, NoneType], post_check_func: Union[Callable, NoneType])[source]¶
- __eq__(other)¶
Return self==value.
- __hash__ = None¶
- __init__(name: str, baremetal_targets: List[str], check_func: Optional[Callable], post_check_func: Optional[Callable]) None ¶
- __repr__()¶
Return repr(self).
- class scap.main.LockManager(exe_name)[source]¶
Holds a lock open for a given repository.
examples:
lock 'Testing something, do not deploy'
- class scap.main.RebuildCdbs(exe_name)[source]¶
Rebuild localization cache CDB files from the JSON versions.
- class scap.main.RefreshCdbJsonFiles(exe_name)[source]¶
Create JSON/MD5 files for all CDB files in a directory.
This will put a JSON and MD5 file in /upstream for each CDB file.
This can be combined with rsync and the scap-rebuild-cdbs to push out large CDB files with minimal traffic. CDB files change drastically with small key/value changes, where as JSON files do not, and thus they diff/rdiff much better.
When pushing updates with rsync, this should be run before running rsync. The rsync command should exclude CDB files or at least use -ignore-existing. After the rsync is done, scap-rebuild-cdbs can be run on each server to apply the updates to the CDB files.
- class scap.main.ScapWorld(*args, **kwargs)[source]¶
Deploy MediaWiki to the cluster.
Validate php syntax of wmf-config and multiversion
Compile wikiversions.json to php in staging directory
Update l10n files in staging area
Compute git version information
Ask scap masters to sync with current master
Ask scap proxies to sync with master server
Ask apaches to sync with fastest rsync server (excluding wikiversions.php)
Ask apaches to rebuild l10n CDB files
Ask apaches to sync wikiversions.php
Run purgeMessageBlobStore.php
Rolling invalidation of all opcache for php 7.x
- class scap.main.SecurityPatchCheck(exe_name)[source]¶
Check if security patches are applied.
class to check if patches in
/srv/patches
have been applied to the active wikiversions
- class scap.main.SyncMaster(exe_name)[source]¶
Sync local MediaWiki staging directory with deploy server state.
- class scap.main.SyncPull(exe_name)[source]¶
Sync local MediaWiki deployment directory with deploy server state.
- class scap.main.SyncWikiversions(exe_name)[source]¶
Rebuild and sync wikiversions.php to the cluster.
Command Execution¶
scap.cmd¶
command execution
Copyright © 2014-2017 Wikimedia Foundation and Contributors.
This file is part of Scap.
Scap is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, version 3.
This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
You should have received a copy of the GNU General Public License along with this program. If not, see <http://www.gnu.org/licenses/>.
- class scap.cmd.Command(*cmds)[source]¶
Command is used to store a parameterized shell command for reuse by cluster_ssh and other utilities. The idea is that we store the command, along with any arguments that need to be filled in, then we can call the command later just like a normal python function, which returns the shell command to be ran.
Usage Example:
import scap.cmd as cmd ssh = cmd.Command('/usr/bin/ssh', cmd.arg('user', '-oUser={}')) sudo = cmd.Command('sudo', cmd.arg('user', '-u {}'), '-n', '--') ssh('some.host', sudo('remote_cmd', 'some', 'args', user='sudo_user'), user='ssh_user') # result: ['/usr/bin/ssh', '-oUser=ssh_user', 'some.host', 'sudo', '-u sudo_user', '-n', '--', 'remote_cmd', 'some', 'args']
- class scap.cmd.arg(name, cmd)[source]¶
Represent a named parameter that will be passed to an instance of Command.
For example,
arg('user', '-u {}')
creates an arg called ‘user’ which will evaluate to"-u luser"
when the command is evaluated withsome_command(user='luser')
- Parameters:
name – The name of the parameter, used to specify it’s value later when the command is evaluated.
cmd – string, the argument text format string.
scap.ssh¶
This module provides functions for running commands on remote hosts via SSH.
Copyright © 2014-2017 Wikimedia Foundation and Contributors.
This file is part of Scap.
Scap is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, version 3.
This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
You should have received a copy of the GNU General Public License along with this program. If not, see <http://www.gnu.org/licenses/>.
- class scap.ssh.JSONOutputHandler(host)[source]¶
Deserialize and log structured JSON output from hosts.
Any non-structured output is stored for future handling.
- class scap.ssh.Job(hosts=None, command=None, user=None, logger=None, key=None, verbose=False)[source]¶
Execute a job on a group of remote hosts via ssh.
- run(batch_size=80, return_jobresults=False)[source]¶
Run the job, report progress, and return success/failed counts or a JobResults object if return_jobresults is truthy.
- Returns:
(ok, failed) counts of successful/failed hosts or a JobResults object if return_jobresults is truthy.
- Raises:
RuntimeError if command has not been set
scap.tasks¶
Contains functions implementing scap tasks
Copyright © 2014-2017 Wikimedia Foundation and Contributors.
This file is part of Scap.
Scap is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, version 3.
This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
You should have received a copy of the GNU General Public License along with this program. If not, see <http://www.gnu.org/licenses/>.
- scap.tasks._call_rebuildLocalisationCache(app, version, out_dir, use_cores=1, php_l10n=False, lang=None, force=False, quiet=False, delay_messageblobstore_purge=False)[source]¶
Helper for update_localization_cache.
- Parameters:
app – Scap cli.Application
version – The train version to build l10n for
out_dir – The output directory
use_cores – The number of cores to run in
lang – The –lang option, or None to omit
force – Whether to pass –force
quiet – Whether to pass –quiet
- scap.tasks.cache_git_info(version, cfg)[source]¶
Create JSON cache files of git branch information.
- Parameters:
version – MediaWiki version (eg ‘1.38.0-wmf.20’)
cfg – Dict of global configuration values
- Raises:
IOError
if version directory is not found
- scap.tasks.check_patch_files(version, cfg)[source]¶
Check to see if there are unmerged patch files from /srv/patches for a given revision.
- Parameters:
version – MediaWiki version string (e.g., ‘1.27.0-wmf.8’)
cfg – Scap configuration dict
- scap.tasks.clear_message_blobs(app, logger=None)[source]¶
Clear MessageBlobStore cache on all wikis
- Parameters:
app – Scap cli.Application
logger – logger instance
- scap.tasks.compile_wikiversions(source_tree, cfg, logger=None)[source]¶
Validate and compile the wikiversions.json file.
Find the realm specific filename for wikiversions.json in specified tree (deploy or staging)
Validate that all versions mentioned in the json exist as directories in the specified tree.
Create a temporary php file from the json contents
Atomically rename the temporary php to the realm specific wikiversions.php filename
- Parameters:
source_tree – Source tree to read file from: ‘deploy’ or ‘stage’
cfg – Dict of global configuration values
- scap.tasks.ensure_extension_messages_file(cfg, version, logger)[source]¶
Ensure that <staging>/wmf-config/ExtensionMessages-<version>.php exists. If it does not exist, create a blank file.
In either case, the path to the ExtensionMessages-<version>.php file is returned.
- scap.tasks.get_wikiversions_ondisk(directory) list [source]¶
Returns a list of the train branch versions that are currently checked out in DIRECTORY. The list is sorted in ascending order.
- Returns:
list like:: [‘1.41.0-wmf.1’, ‘1.41.0-wmf.2’]
- scap.tasks.handle_services(services, require_valid_service=False, on_secondary_host=False)[source]¶
Take a comma-separated list of services and restart, reload or disable each of them.
The idea is to take a string directly from the scap.cfg file that looks like:
jobrunner, jobchron=reload, jenkins=restart:disable-secondary, …
and be able to determine what to do with that list.
If no action is specified the default is “restart”. When specified, main action can be one of:
restart: service will be restarted
reload: service will be reloaded
When specified, secondary action can be one of:
disable-secondary: service is disabled on targets marked as secondary
Note that specifying a secondary action requires a main action
- scap.tasks.merge_cdb_updates(directory, pool_size, trust_mtime=False, mute=False, logger=None)[source]¶
Update l10n CDB files using JSON data.
- Parameters:
directory – L10n cache directory
pool_size – Number of parallel processes to use
trust_mtime – Trust file modification time?
mute – Disable progress indicator
- scap.tasks.refresh_cdb_json_file(cdb_file_path) bool [source]¶
Rebuild json file from cdb file.
Check md5 file saved in upstream against md5 of cdb file
Read cdb file to dict
Write dict to named temporary file
Change permissions on named temporary file
Overwrite upstream json file
Write upstream md5 file
Returns a boolean indicating whether or not a JSON file was (re)created.
- scap.tasks.refresh_cdb_json_files(in_dir, pool_size, verbose)[source]¶
Update json files from corresponding cdb file in parallel.
- Parameters:
in_dir – directory containing cdb files
pool_size – number of “threads” to use
verbose – output verbosely
- scap.tasks.sync_common(app, include=None, sync_from=None, verbose=False, logger=None, rsync_args=None, exclude_wikiversionsphp=False)[source]¶
Sync local deploy dir with upstream rsync server’s copy.
Rsync from
server::common
to the local deploy directory. If a list of servers is given insync_from
we will attempt to select the “best” one to sync from. If no servers are given or all servers given have issues we will fall back to using the server named bymaster_rsync
in the configuration data.- Parameters:
app – cli.Application
include – List of rsync include patterns to limit the sync to. If
None
is given the entirecommon
module on the target rsync server will be transferred. Rsync syntax for syncing a directory is<dirname>/***
.sync_from – List of rsync servers to fetch from.
- scap.tasks.sync_master(app, master, verbose=False, logger=None)[source]¶
Sync local staging dir with upstream rsync server’s copy.
Rsync from
server::common
to the local staging directory.- Parameters:
app – cli.Application
master – Master server to sync with
verbose – Enable verbose logging?
- scap.tasks.sync_wikiversions(hosts, app, key=None)[source]¶
Rebuild and sync wikiversions.php to the cluster.
- Parameters:
hosts – List of hosts to sync to
app – cli.Application
- scap.tasks.update_l10n_cdb(cache_dir, cdb_file, trust_mtime=False, logger=None)[source]¶
Update a localization CDB database.
- Parameters:
cache_dir – L10n cache directory
cdb_file – L10n CDB database
trust_mtime – Trust file modification time?
- scap.tasks.update_l10n_cdb_wrapper(args, logger=None)[source]¶
Wrapper for update_l10n_cdb to be used in contexts where only a single argument can be provided.
- Parameters:
args – Sequence of arguments to pass to update_l10n_cdb
- scap.tasks.update_localization_cache(version, app, logger=None, json=True)[source]¶
Update the localization cache for a given MW version.
- Parameters:
version – MediaWiki version
wikidb – Wiki running given version
app – Application calling the function
json – Whether to generate JSON/MD5 files from CDBs when finished.
Git Repository Deployment¶
scap.deploy¶
Command wrappers for deploy tasks
Copyright © 2014-2017 Wikimedia Foundation and Contributors.
This file is part of Scap.
Scap is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, version 3.
This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
You should have received a copy of the GNU General Public License along with this program. If not, see <http://www.gnu.org/licenses/>.
- class scap.deploy.Deploy(exe_name)[source]¶
Sync new service code across cluster.
- _execute_for_group(stages, group, ignore_failure=False, prompt_user=False)[source]¶
Executes the given stages across targets of the given group’s subgroups.
- Parameters:
stages – List of stages to execute
group – The targets.DeployGroup for which to execute the stages. Any target host that is unreachable via SSH will be added to the group’s list of excluded hosts.
ignore_failure – Whether to keep on rolling past the failure_limit threshold. Note that even with this argument, SSH failure result in the target being excluded from future stage execution.
prompt_user – Whether to prompt the user after each subgroup.
- _load_config(use_global_config=True)[source]¶
Set the host directory after the config has been loaded.
- _needs_latest_sha1(stages)[source]¶
Determine whether we expect a new SHA1 to be tagged and deployed
- Stages:
list of stages being run
- config_deploy_setup(commit)[source]¶
Generate environment-specific config file and variable template list.
Builds a yaml file that contains: 1. A list of file objects containing template files to be deployed 2. An object containing variables specified in the environment-specific vars.yaml file and inheriting from the vars.yaml file
- class scap.deploy.DeployLocal(exe_name)[source]¶
Command that runs on target hosts.
Responsible for fetching code from the git server, checking out the appropriate revisions, restarting/disabling services and running checks.
- static _check_applies(chk, stage, group, when)[source]¶
Verify whether a check applies for the current group and stage.
- _execute_checks(when, stage, group=None, logger=None)[source]¶
Fetch and executes all checks configured for the given stage.
Checks are retrieved from the remote deploy host and cached within tmp.
- _get_config_overrides()[source]¶
Get config information locally or from the deployment host.
If the local [repo]-cache/.config file does not exist, or –refresh-config has been passed explicitly on the command line, then the config is fetched from the deployment server. Otherwise the local config cache is used.
This is useful for things like locally rebuilding config files when a remote var file has changed, and the version deployed from the deployment server has changed, but a particular target has not yet been updated.
- config_deploy()[source]¶
Render config files.
Grabs the current config yaml file from the deploy git server, and renders the config files templates inside the repo-cache’s .git/config-files directory. During the promote stage, final links are created to the rev-specific config files.
If the config file target path for a template is relative, however, the file is written directly at the target path and no link is created.
- config_diff()[source]¶
Render config files from DEPLOY_HEAD and compare each file to the deployed version. This is called by scap deploy –dry-run
- fetch()[source]¶
Fetch the specified revision of the remote repo.
The given repo is cloned into the cache directory and a new working directory for the given revision is created under revs/{rev}.
At the end of this stage, the .in-progress link is created to signal the possibility for future rollback.
- finalize(rollback=False, rev=None)[source]¶
Perform the final deploy actions.
Moves the .done flag to the rev directory, removes the .in-progress flag, and cleans up old revision directories.
- handle_service()[source]¶
Restart, reload or disable service(s) and check port based on configuration.
- main(*extra_args)[source]¶
Main business logic of the application.
Parsed command line arguments are available in self.arguments. Global configuration is available in self.config. Unparsed command line arguments are passed as positional arguments.
- Returns:
exit status
- promote(rev=None, rev_dir=None, config_deploy=True)[source]¶
Promote the current deployment.
Switches the current symlink to the current revision directory and restarts the configured service.
Probes the configured service port to measure whether it successfully restarted.
- rollback()[source]¶
Performs a rollback to the last deployed revision.
Rollback looks for a .done symlink that points to the revision directory for the last successful deployment. If this link doesn’t exist, a rollback isn’t possible. If it does exist, the current revision directory is replaced with the target of the link and the promote and finalize stages are re-run.
- class scap.deploy.DeployLog(exe_name)[source]¶
Tail/filter/output events from the deploy logs.
examples:
deploy-log -v deploy-log 'host == scap-target-01' deploy-log 'msg ~ "some important (message|msg)"' deploy-log 'levelno >= WARNING host == scap-target-*'
scap.git¶
Helpers for git operations and interacting with .git directories
- scap.git.clone_or_update_repo(dir, repo, branch, logger, reference=None, ref=None)[source]¶
Clone or update the checkout of ‘repo’ in ‘dir’, using the specified branch. Note that existing repos are hard-reset to the match the state of the origin.
Submodules are handled as well.
Returns the sha1 of the checked out branch.
- scap.git.describe(location)[source]¶
Returns a convenient label for the current state of the git repo.
- scap.git.fat_isinitialized(location)[source]¶
Returns whether git-fat has been initialized for the given directory.
- scap.git.fetch(location, repo, reference=None, dissociate=True, recurse_submodules=False, shallow=False, bare=False, config=None, branch=None)[source]¶
Fetch a git repo to a location
- scap.git.file_has_unstaged_changes(file, location=None) bool [source]¶
Untracked files are also considered
- scap.git.get_branch(directory)[source]¶
Returns the name of the current branch in the git repo located in directory
- scap.git.get_config(key)[source]¶
Returns the value of the specified key in git global config (i.e. ~/.gitconfig) if found, otherwise returns None.
- scap.git.info(directory, remote='origin')[source]¶
Compute git version information for a given directory that is compatible with MediaWiki’s GitInfo class.
- Parameters:
directory – Directory to scan for git information
- Returns:
Dict of information about current repository state
- scap.git.info_filename(directory, install_path, cache_path)[source]¶
Compute the path for a git_info cache file related to a given directory.
>>> info_filename('foo', 'foo', '') 'info.json' >>> info_filename('foo/bar/baz', 'foo', 'xyzzy') 'xyzzy/info-bar-baz.json'
- scap.git.is_dir(path, top_level=False) bool [source]¶
Returns True if ‘path’ is a git checkout directory, False otherwise.
If ‘top_level’ is true, ‘path’ must be the top level of a git checkout.
- scap.git.largefile_pull(location, implementor, submodules=False)[source]¶
Syncs all git-fat or git-lfs objects for the given repo directory.
- Parameters:
location – Repository to work in
implementor – What implementation to pull with (git-lfs, git-fat)
- scap.git.list_submodules_paths_urls(repo, args)[source]¶
Return a list of the paths and URLs of the submodules of the given repository, separated by a space
- scap.git.next_deploy_tag(location)[source]¶
Calculates the scap/sync/YYYY-MM-DD/{n} tag to use for this deployment
- scap.git.parse_submodules(location) dict [source]¶
Reads .gitmodules and returns a dictionary of information about each defined submodule.
The key is the name of the submodule, and the value is a dictionary with “path” (typically the same as the submodule name) and “url” keys.
- scap.git.remap_submodules(location, server)[source]¶
Remap all submodules to deployment server
This function supports remapping submodules available on the deployment server. Since the remote is a non-bare repo (as of git 1.7.8) all submodules should be available over http under the remote server checkout’s git dir: [server]/[repo]/.git/modules/[submodule_path]
- Parameters:
location – String path to local git checkout containing a .gitmodules file
server – String path to remote, non-bare, repo gitdir
- scap.git.remote_set_url(location, url, remote='origin', push=False)[source]¶
In the git repo at ‘location’, set the url of the specified remote.
- scap.git.resolve_gitdir(directory)[source]¶
Find the .git directory for a given path.
This will resolve the gitlink if path/.git is a gitlink to a bare repo e.g. a file with one line, like this:
gitdir:/path/to/somewhere/else
- scap.git.update_deploy_head(deploy_info, location)[source]¶
updates .git/DEPLOY_HEAD file
- Parameters:
deploy_info – current deploy info to write to file as YAML
location ((optional)) – git directory location (default cwd)
scap.context¶
Management of deployment host/target directories and execution context.
Copyright © 2014-2017 Wikimedia Foundation and Contributors.
This file is part of Scap.
Scap is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, version 3.
This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
You should have received a copy of the GNU General Public License along with this program. If not, see <http://www.gnu.org/licenses/>.
- class scap.context.Context(root, environment=None)[source]¶
Base context for either the deployment host or target.
- class scap.context.HostContext(root, environment=None)[source]¶
Manage deployment host paths and execution context.
- env_specific_path(*relpaths)[source]¶
Return path to default or environment specific file/directory.
Both the environment specific path at
scap/environments/{name}
and the default path atscap
is searched in respective order. The first path that exists will be returned.
- class scap.context.TargetContext(root, environment=None)[source]¶
Manage target host paths and execution context.
- property cache_dir¶
Path to the cached repo clone.
- property current_link¶
Symlink that points to the currently deployed revision.
- property current_rev_dir¶
Real path to the currently deployed revision.
- property done_rev_dir¶
Real path to the revision previously marked as done.
- find_old_rev_dirs(cache_revs=5)[source]¶
Generate revision directories that are candidates for deletion.
The cache_revs most recent revision directories and any revision directory that is current or in progress is not considered.
- link_path_to_rev(path, rev, backup=False)[source]¶
Create link to the given rev’s directory at the given path.
- property local_config¶
Local target file that has a copy of the last-deployed config.
- mark_rev_current(rev)[source]¶
Change the current rev to the given one.
This state is maintained as a
current
symlink in the directory root that points to the relevantrevs/{rev}
directory.
- mark_rev_done(rev)[source]¶
Change the state to done for the given rev.
This state is maintained as a
.done
symlink in the directory root that points to the relevantrevs/{rev}
directory.
- mark_rev_in_progress(rev)[source]¶
Change the state to in-progress for the given rev.
This state is maintained as a
.in-progress
symlink in the directory root that points to the relevantrevs/{rev}
directory.
- property rev_done¶
The rev that is currently marked as done.
- property rev_in_progress¶
The rev that is currently marked as in-progress.
- property revs_dir¶
Context directory that stores revisions.
General Utilities¶
scap.utils¶
Contains misc utility functions.
- class scap.utils.VarDumpJSONEncoder(*, skipkeys=False, ensure_ascii=True, check_circular=True, allow_nan=True, sort_keys=False, indent=None, separators=None, default=None)[source]¶
encode python objects to json
- default(o)[source]¶
Implement this method in a subclass such that it returns a serializable object for
o
, or calls the base implementation (to raise aTypeError
).For example, to support arbitrary iterators, you could implement default like this:
def default(self, o): try: iterable = iter(o) except TypeError: pass else: return list(iterable) # Let the base class default method raise the TypeError return JSONEncoder.default(self, o)
- scap.utils.cd(dirname)[source]¶
Context manager. Cds to dirname.
It moves back to previous dir on context exit. :param dirname: directory into which it should change
- scap.utils.context_logger(context_name, *args)[source]¶
Context manager that maintains nested logger contexts.
Each time you enter a with block using this context manager, a named logger is set up as a child of the current logger. When exiting the with block, the logger gets popped off the stack and the parent logger takes it’s place as the ‘current’ logging context.
The easiest way to use this is to decorate a function with log_context, For Example:
@log_context('name') def my_func(some, args, logger=None): logger.debug('something')
- scap.utils.dir_is_empty(path)[source]¶
Returns True if ‘path’ is an empty directory, otherwise returns False. Raises an error if ‘path’ does not name a directory.
- scap.utils.eintr_retry(func, *args)[source]¶
Retry a system call if it is interrupted by EINTR.
Extracted from stdlib’s subprocess (where it is called _eintr_retry_call – the leading underscore indicating it is not part of the module’s API). This is not needed on Python >= 3.5, thanks to PEP 0475.
- scap.utils.find_nearest_host(hosts, port=22, timeout=1)[source]¶
Given a collection of hosts, find the one that is the fewest number of hops away.
>>> # Begin test fixture >>> import socket >>> fixture_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM) >>> fixture_socket.bind(('127.0.0.1', 0)) >>> fixture_socket.listen(1) >>> fixture_port = fixture_socket.getsockname()[1] >>> # End test fixture >>> find_nearest_host(['127.0.0.1'], port=fixture_port) '127.0.0.1'
- Parameters:
hosts – Hosts to check
port – Port to try to connect on (default: 22)
timeout – Timeout in seconds (default: 1)
- scap.utils.find_regular_files(dirname)[source]¶
Return sorted list of all regular files under a directory
- scap.utils.find_upwards(name, starting_point='/srv/app')[source]¶
Search the specified directory, and all parent directories, for a given filename, returning the first matching path that is found.
- Parameters:
name – the relative path name to search for
starting_point – the directory to start searching from. Each parent directory will be searched until a match is found.
- Returns:
if a match is found, returns the full path to the matching file, None otherwise.
- scap.utils.get_active_wikiversions(directory, realm, return_type=<class 'list'>)[source]¶
Get an ordered collection of active MediaWiki versions.
- Returns:
If ‘return_type’ is list (the default), returns a list of versions, sorted in ascending order.
If ‘return_type’ is dict (the default), returns a collections.OrderedDict of {version:wikidb} values sorted by version number in ascending order. ‘wikidb’ will be the alphabetically-first wikidb for ‘version’. This can be used by operations that need a db but don’t care which wiki’s db is used.
- scap.utils.get_current_train_info(api_url, proxy=None) dict [source]¶
Returns a dictionary containing information about this week’s train
- scap.utils.get_current_train_version_from_gerrit(gerrit_url) str [source]¶
Returns a string like ‘1.39.0-wmf.19’
- scap.utils.get_env_specific_filename(path, env=None)[source]¶
Find a file specific to the environment in which scap is running.
- scap.utils.get_group_versions(group, directory, realm) list [source]¶
Returns a list of versions used by ‘group’, in ascending version order.
- scap.utils.get_patches(sub_dirs, root_dir)[source]¶
Find all patches under each subdirectory.
:param sub_dirs list of sub directories under which to search :param root_dir base path under which subdirectories reside :return dictionary of patches, keyed by sub_dir
- scap.utils.get_realm_specific_filename(filename, realm)[source]¶
If a realm-specific version of ‘filename’ exists, return it, otherwise return ‘filename’. To construct the realm-specific filename, “-REALM” is inserted before the file extension. For example, “wikiversions.json” becomes “wikiversions-REALM.json”.
- scap.utils.get_train_blockers_info(api_url, proxy=None) dict [source]¶
Returns a dictionary with details about the current and upcoming train blocker tasks
- scap.utils.human_duration(elapsed)[source]¶
Format an elapsed seconds count as human readable duration.
>>> human_duration(1) '00m 01s' >>> human_duration(65) '01m 05s' >>> human_duration(60*30+11) '30m 11s'
- scap.utils.is_phabricator_task_id(string: str) bool [source]¶
Returns true if ‘string’ has the format of a phabricator task id
- scap.utils.isclose(a, b, rel_tol=1e-09, abs_tol=0.0)[source]¶
Return True if a is close in value to b. False otherwise.
- Parameters:
a – one of the values to be tested
b – the other value to be tested
rel_tol=1e-9 – The relative tolerance – the amount of error allowed, relative to the absolute value of the larger input values.
abs_tol=0.0 – The minimum absolute tolerance level – useful for comparisons to zero.
Copyright: Christopher H. Barker Original License: Apache License 2.0 <https://github.com/PythonCHB/close_pep>
- scap.utils.iterate_subdirectories(root)[source]¶
Generator over the child directories of a given directory.
- scap.utils.join_path(*fragments)[source]¶
Join several path fragments into a complete, normalized path string.
Strips leading and trailing slashes from path fragments to avoid an unfortunate feature of os.path.join() which is described in the python documentation for os.path as follows:
“If any component is an absolute path, all previous components are thrown away, and joining continues.”
- scap.utils.list_intersection(list1, list2)[source]¶
Returns a list containing the intersection (items in common) of list1 and list2
- scap.utils.log_context(context_name)[source]¶
Decorator to wrap the a function in a new context_logger.
The logger is passed to the function via a kwarg named ‘logger’.
- scap.utils.make_sudo_check_call_env(env)[source]¶
Returns a string of environment variables formatted for the shell
sudo 1.8.21 adds support for adding a list of variables to –preserve-env, that should replace this function in future
- scap.utils.md5_file(path)[source]¶
Compute the md5 checksum of a file’s contents.
- Parameters:
path – Path to file
- Returns:
hexdigest of md5 checksum
- scap.utils.mkdir_p(path)[source]¶
Create directory path.
- Parameters:
path – The directory path to be created.
- scap.utils.open_with_lock(path, mode='r', *args, **kwargs)[source]¶
Opens the given file and acquires an advisory lock using the open file object. If the mode is read-only (‘r’ or ‘rb’), the lock is acquired as shared, and otherwise acquired as exclusive.
- scap.utils.ordered_load(stream, Loader=<class 'yaml.loader.SafeLoader'>, object_pairs_hook=<class 'collections.OrderedDict'>)[source]¶
Load yaml files and keeping order.
From stackoverflow.com/questions/5121931
:param stream the file object to read :param loader yaml.SafeLoader or its subclasses :object_pairs_hook type of return :return OrderedDict object with the same order of the yaml file
- scap.utils.parse_rsync_stats(string: str) dict [source]¶
Scans the string looking for text like the following and returns a dictionary with the extracted integer fields.
Note that if no such matching text is found an empty dictionary will be returned.
Number of files: 184,935 (reg: 171,187, dir: 13,596, link: 152) Number of created files: 0 Number of deleted files: 0 Number of regular files transferred: 1 Total file size: 8,756,954,367 bytes Total transferred file size: 815,772 bytes Literal data: 0 bytes Matched data: 815,772 bytes File list size: 4,744,396 File list generation time: 0.517 seconds File list transfer time: 0.000 seconds Total bytes sent: 5,603 Total bytes received: 4,744,454
- scap.utils.parse_wmf_version(version: str) Version [source]¶
Parses a string like “1.29.0-wmf.4” and returns a packaging.version.Version object representing the version. These objects can be compared using <, <=, >, >=, ==. The special case version of “master” will be treated as a very large version number. “branch_cut_pretest” and “next” are second-to-highest.
- scap.utils.pluralize(word: str, quantity) str [source]¶
If ‘quantity’ represents a quantity of one, returns ‘word’, otherwise, returns the pluralized version of ‘word’.
‘quantity’ can be an int, or an object that works with len().
- scap.utils.read_first_line_from_file(filename) str [source]¶
Reads and returns the first line of the specified file. Whitespace is stripped.
- scap.utils.read_wikiversions(directory, realm) dict [source]¶
Return a dictionary representing the contents of the realm-specific wikiversions.json file in the specified directory.
Keys are wikidbs, values are “php-<version>”
- scap.utils.select_latest_patches(patch_base_dir)[source]¶
Find the latest /srv/patches/<version> directory. Useful to e.g. populate the patches dir for a new version by carrying over the most recent patches
Returns None if unavailable.
- scap.utils.string_to_base64_string(input: str) str [source]¶
Returns a string representing the base64 encoding of the input string.
- scap.utils.sudo_check_call(user, cmd, logger=None, logLevel=10, app=None)[source]¶
Run a command as a specific user.
Reports stdout/stderr of process to logger during execution.
Returns a string containing stdout/stderr output from the subprocess.
- Parameters:
user – User to run command as
cmd – Command to execute
logger – Logger to send process output to
app – Application calling the function, required if the command is scap
- Raises:
subprocess.CalledProcessError on non-zero process exit
- Raises:
ValueError if the command is scap and app was not specified
- scap.utils.sudo_temp_dir(owner, prefix)[source]¶
Create a temporary directory and delete it after the block.
- Parameters:
owner – Directory owner
prefix – Temp directory prefix
- Returns:
Full path to temporary directory
- scap.utils.suppress_backtrace()[source]¶
Context manager that sets the “don’t backtrace” flag on any exception that occurs within context.
Can be overridden by setting the environment variable SCAP_BACKTRACE.
- Example:
- def my_function():
- with suppress_backtrace():
some_function_that_may_reasonably_fail()
- scap.utils.temp_to_permanent_file(final_filename, mode='w')[source]¶
temp_to_permanent_file yields (by default) a text stream on a temporary file that is open for writing. If the context body completes without exception, the temp file is renamed to final_filename, atomically replacing any existing file of that name. If an exception is raised during the exception of the body, the temp file is deleted and final_filename remains unaffected.
Specify mode=”wb” for a binary stream.
Example:
- with temp_to_permanent_file(“/tmp/important”) as f:
f.write(“Important information”)
- scap.utils.valid_version(version: str) bool [source]¶
Returns True if version is a valid version. In this context valid means suitable for naming a <staging dir>/php-<version> directory. This means that “auto” is not an acceptable version here.
- scap.utils.version_argument_parser(ver: str, allow_auto=True) str [source]¶
Validate a mediawiki version number argument
- scap.utils.write_file_if_needed(filename, data: str) bool [source]¶
Write ‘data’ to ‘filename’ if ‘filename’ doesn’t already have that data in it
When updating is needed, the file is replaced atomically by writing to a temp file and then renaming to the final name.
Note, the file is written in text mode.
Returns a boolean indicating whether or not the file was updated.
scap.ansi¶
ANSI escape codes
..seealso:: https://en.wikipedia.org/wiki/ANSI_escape_code
Copyright © 2014-2017 Wikimedia Foundation and Contributors.
This file is part of Scap.
Scap is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, version 3.
This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
You should have received a copy of the GNU General Public License along with this program. If not, see <http://www.gnu.org/licenses/>.
- scap.ansi.esc(*args)[source]¶
Get an ANSI escape code.
>>> esc(BG_WHITE, FG_RED, BLINK) == r'[5;31;47m' True
- Parameters:
args – ANSI attributes
- Returns:
str
- scap.ansi.format_ansi(*args)[source]¶
create an ansi color string from a list of color codes and plain strings.
>>> format_ansi((FG_BLUE,BG_WHITE),'blue on white') == '[34;47mblue on white[0m' True
- Parameters:
args – ANSI color codes and strings of text
- Returns:
str
- scap.ansi.logo(eyes=None, color=True, **colors)[source]¶
Get the scap logo.
Scappy the scap pig:
___ ____ ⎛ ⎛ ,---- \ //==--' _//| .·//==--' ____________________________ _OO≣=- ︶ ᴹw ⎞_§ ______ ___\ ___\ ,\__ \/ __ \ (∞)_, ) ( | ______/__ \/ /__ / /_/ / /_/ / ¨--¨|| |- ( / _______\____/\___/ \__^_/ .__/ ««_/ «_/ jgs/bd808 /_/
Ascii art derived from original work by Joan Stark [1] and the speed figlet font [2].
- Parameters:
color – Color logo using ANSI escapes
colors – Alternate colors
- Returns:
str
scap.template¶
Module for working with file templates
Copyright © 2014-2017 Wikimedia Foundation and Contributors.
This file is part of Scap.
Scap is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, version 3.
This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
You should have received a copy of the GNU General Public License along with this program. If not, see <http://www.gnu.org/licenses/>.
- class scap.template.Template(name, loader, erb_syntax=False, var_file=None, overrides=None, output_format=None)[source]¶
Adapter class that wraps jinja2 templates.
- __init__(name, loader, erb_syntax=False, var_file=None, overrides=None, output_format=None)[source]¶
Logging and Monitoring¶
scap.log¶
Helpers for routing and formatting log data.
Copyright © 2014-2017 Wikimedia Foundation and Contributors.
This file is part of Scap.
Scap is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, version 3.
This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
You should have received a copy of the GNU General Public License along with this program. If not, see <http://www.gnu.org/licenses/>.
- class scap.log.AnsiColorFormatter(fmt=None, datefmt=None, colors=None, colorize='auto')[source]¶
Colorize output according to logging level.
- __init__(fmt=None, datefmt=None, colors=None, colorize='auto')[source]¶
- Parameters:
fmt – Message format string
datefmt – Time format string
colors – Dict of {‘levelname’: ANSI SGR parameters}
colorize – Set to true to colorize messages based on log level. Default ‘auto’ enable color when stderr is a tty or ‘FORCE_COLOR’ is found in the environment.
- format(record)[source]¶
Format the specified record as text.
The record’s attribute dictionary is used as the operand to a string formatting operation which yields the returned string. Before formatting the dictionary, a couple of preparatory steps are carried out. The message attribute of the record is computed using LogRecord.getMessage(). If the formatting string uses the time (as determined by a call to usesTime(), formatTime() is called to format the event time. If there is exception information, it is formatted using formatException() and appended to the message.
- class scap.log.DeployLogFormatter(fmt=None, datefmt=None, style='%')[source]¶
Ensure that all deploy.log records contain a host attribute.
- format(record)[source]¶
Format the specified record as text.
The record’s attribute dictionary is used as the operand to a string formatting operation which yields the returned string. Before formatting the dictionary, a couple of preparatory steps are carried out. The message attribute of the record is computed using LogRecord.getMessage(). If the formatting string uses the time (as determined by a call to usesTime(), formatTime() is called to format the event time. If there is exception information, it is formatted using formatException() and appended to the message.
- class scap.log.DiffLogFormatter(fmt=None, datefmt=None, colors=None, colorize='auto')[source]¶
- __init__(fmt=None, datefmt=None, colors=None, colorize='auto')[source]¶
- Parameters:
fmt – Message format string
datefmt – Time format string
colors – Dict of {‘levelname’: ANSI SGR parameters}
colorize – Set to true to colorize messages based on log level. Default ‘auto’ enable color when stderr is a tty or ‘FORCE_COLOR’ is found in the environment.
- format(record)[source]¶
Format the specified record as text.
The record’s attribute dictionary is used as the operand to a string formatting operation which yields the returned string. Before formatting the dictionary, a couple of preparatory steps are carried out. The message attribute of the record is computed using LogRecord.getMessage(). If the formatting string uses the time (as determined by a call to usesTime(), formatTime() is called to format the event time. If there is exception information, it is formatted using formatException() and appended to the message.
- class scap.log.Filter(criteria, invert=True)[source]¶
Generic log filter that matches record attributes against criteria.
You can provide either a glob pattern, regular expression, or lambda as each attribute criterion, and invert the logic by passing filter=False.
Examples:
Filter({'name': '*.target.*', 'host': 'scap-target-01'}) Filter({'msg': re.compile('some annoying (message|msg)')}) Filter({'levelno': lambda lvl: lvl < logging.WARNING}) Filter({'name': '*.target.*'}, invert=False)
Equivalent DSL examples:
Filter.loads('name == *.target.* host == scap-target-01') Filter.loads('msg ~ "some annoying (message|msg)"') Filter.loads('levelno < WARNING')f Filter.loads('name == *.target.*', invert=False)
- append(criteria)[source]¶
Append the filter with the given criteria.
- Parameters:
criteria (iter) – Filter criteria
- filter(record)[source]¶
Perform filtering on a given log record.
- Parameters:
record (LogRecord) – Log record.
- class scap.log.IRCSocketHandler(host, port, timeout=1.0)[source]¶
Log handler for logmsgbot on #wikimedia-operation.
Sends log events to a tcpircbot server for relay to an IRC channel.
- class scap.log.JSONFormatter(fmt=None, datefmt=None, style='%')[source]¶
Serialize logging output as JSON.
Can be used to maintain logged event structure between the deployment host and remote targets.
- format(record)[source]¶
Format the specified record as text.
The record’s attribute dictionary is used as the operand to a string formatting operation which yields the returned string. Before formatting the dictionary, a couple of preparatory steps are carried out. The message attribute of the record is computed using LogRecord.getMessage(). If the formatting string uses the time (as determined by a call to usesTime(), formatTime() is called to format the event time. If there is exception information, it is formatted using formatException() and appended to the message.
- class scap.log.LogstashFormatter(fmt=None, datefmt='%Y-%m-%dT%H:%M:%SZ', log_type='scap')[source]¶
Format log messages for logstash.
- __init__(fmt=None, datefmt='%Y-%m-%dT%H:%M:%SZ', log_type='scap')[source]¶
- Parameters:
fmt – Message format string (not used)
datefmt – Time format string
type – Logstash event type
- converter()¶
- gmtime([seconds]) -> (tm_year, tm_mon, tm_mday, tm_hour, tm_min,
tm_sec, tm_wday, tm_yday, tm_isdst)
Convert seconds since the Epoch to a time tuple expressing UTC (a.k.a. GMT). When ‘seconds’ is not passed in, convert the current time instead.
If the platform supports the tm_gmtoff and tm_zone, they are available as attributes only.
- class scap.log.MuteReporter(name='', expect=0, fd=<_io.TextIOWrapper name='<stderr>' mode='w' encoding='utf-8'>)[source]¶
A report that declines to report anything.
- class scap.log.ProgressReporter(name, expect=0, fd=<_io.TextIOWrapper name='<stderr>' mode='w' encoding='utf-8'>, spinner=None)[source]¶
Track and display progress of a process.
Report on the status of a multi-step process by displaying the completion percentage and succes, failure and remaining task counts on a single output line.
- __init__(name, expect=0, fd=<_io.TextIOWrapper name='<stderr>' mode='w' encoding='utf-8'>, spinner=None)[source]¶
- Parameters:
name – Name of command being monitored
expect – Number of results to expect
fd – File handle to write status messages to
spinner – Cyclical iterator that returns progress spinner.
- class scap.log.QueueReporter(name, expect=0, fd=<_io.TextIOWrapper name='<stderr>' mode='w' encoding='utf-8'>, queue=None)[source]¶
A ProgressReporter which sends its state-changing operations upstream via a queue. It does not generate any output.
- __init__(name, expect=0, fd=<_io.TextIOWrapper name='<stderr>' mode='w' encoding='utf-8'>, queue=None)[source]¶
- Parameters:
name – Name of command being monitored
expect – Number of results to expect
fd – File handle to write status messages to
spinner – Cyclical iterator that returns progress spinner.
- class scap.log.RateLimitedProgressReporter(*args, **kwargs)[source]¶
The same as ProgressReporter, but doesn’t generate output more than once every max_reporting_interval seconds. The final progress report is always generated.
- class scap.log.Stats(host, port, logger=None)[source]¶
A simple StatsD metric client.
It can log measurements and counts to a remote StatsD host. See <https://github.com/etsy/statsd/wiki/Protocol> for details.
- class scap.log.Timer(description, name='unsupplied', stats=None, logger=None)[source]¶
Context manager to track and record the time taken to execute a block.
Elapsed time will be recorded to a logger and optionally a StatsD server.
>>> with Timer('example'): ... time.sleep(0.1)
>>> s = Stats('127.0.0.1', 2003) >>> with Timer('example', stats=s): ... time.sleep(0.1)
- __init__(description, name='unsupplied', stats=None, logger=None)[source]¶
- Parameters:
description (str) – The human friendly description of the operation being timed.
name (str|None) –
The measurement name to use when transmitting the operation time the stats recording system.
If name is not supplied, the description will be used as the measurement name.
If None, no measurement will be transmitted.
Note that the measurement name will have “scap.” prefixed to it and any non-word characters in the name will be replaced with underscores.
stats (scap.log.Stats) – StatsD client to record block invocation and duration
- class scap.log.Udp2LogHandler(host, port, prefix='scap')[source]¶
Log handler for udp2log.
- __init__(host, port, prefix='scap')[source]¶
- Parameters:
host – Hostname or ip address
port – Port
prefix – Line prefix (udp2log destination)
- makePickle(record)[source]¶
Format record as a udp2log packet.
>>> Udp2LogHandler('127.0.0.1', 12345).makePickle( ... logging.makeLogRecord({'msg':'line1\nline2'})) 'scap line1\nscap line2\n' >>> Udp2LogHandler('127.0.0.1', 12345).makePickle( ... logging.makeLogRecord({'msg':'%s12'% ('0'*65500)})) ... 'scap 00000...00001\n'
- scap.log.log_large_message(message, logger, log_level)[source]¶
Logs ‘message’ to ‘logger’ at the specified ‘log_level’. ‘message’ is broken into multiple messages if it exceeds MAX_MESSAGE_SIZE.
- scap.log.pipe(logger=None, level=20)[source]¶
Yields a write-only file descriptor that forwards lines to the given logger.
- scap.log.reporter(message, mute=False)[source]¶
Instantiate progress reporter
- Message:
string that will be displayed to user
- scap.log.setup_loggers(cfg, console_level=20, handlers=None)[source]¶
Setup the logging system.
Configure the root logger to use
DiffLogFormatter
Optionally add a
Udp2LogHandler
to send logs to a udp2log serverOptional add a
IRCSocketHandler
for the scap.announce log channel to send messages to a tcpircbot server
- Parameters:
cfg – Dict of global configuration values
console_level – Logging level for the local console appender
handlers – Additional handlers
scap.checks¶
Deployment checks.
Definitions are typically loaded from YAML of the following format:
- checks:
- some_unique_check_name:
type: command command: /usr/local/bin/my_special_check stage: promote
- some_other_check_name:
type: nrpe command: some_parsed_nrpe_command_name stage: promote
Copyright © 2014-2017 Wikimedia Foundation and Contributors.
This file is part of Scap.
Scap is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, version 3.
This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
You should have received a copy of the GNU General Public License along with this program. If not, see <http://www.gnu.org/licenses/>.
- class scap.checks.Check(name, stage=None, before=None, after=None, environment=None, group=None, timeout=30.0, command='', shell=False, **opts)[source]¶
Represent a loaded ‘command’ check.
- Parameters:
name – check name
stage – (deprecated: use “after”, ignored when “after” is set) stage after which to run the check
before – stage before which to run the check
after – stage after which to run the check
environment – environment in which to run checks
group – deploy group for which to run the check
timeout – maximum time allowed for check execution, in seconds
command – check command to run
shell – If True, a shell is used to execute “command”.
- class scap.checks.CheckJob(check)[source]¶
Represent and control a running check.
A
CheckJob
begins execution immediately and should be controlled within some kind of poll loop, typically checks.execute.
- scap.checks.checktype(type_of_check)[source]¶
Class decorator for registering a new check type.
- Parameters:
type_of_check – type name
- scap.checks.execute(checks, logger, concurrency=1)[source]¶
Execute the given checks in parallel.
- Parameters:
checks – iterable of checks.Check objects, or a single checks.Check object.
logger – logging.Logger to send messages to
concurrency – level of concurrency
- Returns:
tuple of the aggregate check success and list of executed checks
- Return type:
(bool, list)
scap.nrpe¶
NRPE based deployment checks.
Available Icinga/NRPE command definitions must be loaded and registered for use using load or load_directory, and register, before they can be used in checks.yaml configuration.
- Example checks.yaml:
- checks:
- service_endpoints:
type: nrpe command: check_service_endpoints stage: promote
Copyright © 2014-2017 Wikimedia Foundation and Contributors.
This file is part of Scap.
Scap is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, version 3.
This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
You should have received a copy of the GNU General Public License along with this program. If not, see <http://www.gnu.org/licenses/>.
- class scap.nrpe.NRPECheck(name, stage=None, before=None, after=None, environment=None, group=None, timeout=30.0, command='', shell=False, **opts)[source]¶
Represent a loaded ‘nrpe’ check.
- scap.nrpe.load(config)[source]¶
Load NRPE command definitions from the given configuration.
- Parameters:
config – NRPE configuration string
- Yields:
(name, command)
Third Party¶
cdblib¶
Manipulate DJB’s Constant Databases. These are 2 level disk-based hash tables that efficiently handle many keys, while remaining space-efficient.
When generated databases are only used with Python code, consider using hash() rather than djb_hash() for a tidy speedup.
Note
Minor alterations made to comply with PEP8 style check and to remove attempt to import C implementation of djb_hash. – bd808, 2014-03-04
- scap.cdblib.DJB_HASH(s)¶
Return the value of DJB’s hash function for the given 8-bit string.
>>> py_djb_hash('') 5381 >>> py_djb_hash('') 177572 >>> py_djb_hash('€') 193278953
- scap.cdblib.READ_2_LE4(buffer, /)¶
Return a tuple containing unpacked values.
Unpack according to the format string Struct.format. The buffer’s size in bytes must be Struct.size.
See help(struct) for more on format strings.
- class scap.cdblib.Reader(data, hashfn=<function py_djb_hash>)[source]¶
A dictionary-like object for reading a Constant Database.
Reader accesses through a string or string-like sequence such as mmap.mmap().
- scap.cdblib.WRITE_2_LE4()¶
S.pack(v1, v2, …) -> bytes
Return a bytes object containing values v1, v2, … packed according to the format string S.format. See help(struct) for more on format strings.
- class scap.cdblib.Writer(fp, hashfn=<function py_djb_hash>)[source]¶
Object for building new Constant Databases, and writing them to a seekable file-like object.
- __init__(fp, hashfn=<function py_djb_hash>)[source]¶
Create an instance writing to a file-like object and hash keys.
It uses hashfn to hash keys.
>>> import tempfile >>> temp_fp = tempfile.TemporaryFile() >>> Writer(fp=temp_fp, hashfn=py_djb_hash) <scap.cdblib.Writer object at 0x...>