Python API

• Module Index

Core Application Infrastructure

scap

Wikimedia’s MediaWiki deployment script. Deploys MediaWiki code and configuration to a group of servers via SSH and rsync.

Copyright © 2014-2017 Wikimedia Foundation and Contributors.

This file is part of Scap.

Scap is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, version 3.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program. If not, see <http://www.gnu.org/licenses/>.

class scap.CompileWikiversions(exe_name)

Compile wikiversions.json to wikiversions.php.

main(*extra_args)

Main business logic of the application.

Parsed command line arguments are available in self.arguments. Global configuration is available in self.config. Unparsed command line arguments are passed as positional arguments.

Returns

exit status

class scap.Deploy(exe_name)

Sync new service code across cluster.

__init__(exe_name)

_build_deploy_groups()

Build server groups based on configuration server_groups variable

_execute_for_group(stages, group, ignore_failure=False, prompt_user=False)

Executes the given stages across targets of the given group’s subgroups.

Parameters

• stages – List of stages to execute

• group – The targets.DeployGroup for which to execute the stages. Any target host that is unreachable via SSH will be added to the group’s list of excluded hosts.

• ignore_failure – Whether to keep on rolling past the failure_limit threshold. Note that even with this argument, SSH failure result in the target being excluded from future stage execution.

• prompt_user – Whether to prompt the user after each subgroup.

_get_stage_name(stage)

Map a stage name to a stage display name.

_load_config(use_global_config=True)

Set the host directory after the config has been loaded.

_needs_latest_sha1(stages)

Determine whether we expect a new SHA1 to be tagged and deployed

Stages

list of stages being run

_setup_loggers()

Set up additional logging to scap/deploy.log.

checks_setup()

Build info to run checks.

config_deploy_setup(commit)

Generate environment-specific config file and variable template list.

Builds a yaml file that contains: 1. A list of file objects containing template files to be deployed 2. An object containing variables specified in the environment-specific vars.yaml file and inheriting from the vars.yaml file

execute_stage_on_group(stage, group, targets)

Execute a deploy stage for the given group targets.

Parameters

• stage – deploy stage.

• group – deploy group.

• targets – group targets.

Yields

(host, status)

main(*extra_args)

Main business logic of the application.

Parsed command line arguments are available in self.arguments. Global configuration is available in self.config. Unparsed command line arguments are passed as positional arguments.

Returns

exit status

class scap.DeployLocal(exe_name)

Command that runs on target hosts.

Responsible for fetching code from the git server, checking out the appropriate revisions, restarting/disabling services and running checks.

__init__(exe_name)

static _check_applies(chk, stage, group, when)

Verify whether a check applies for the current group and stage.

_execute_checks(when, stage, group=None, logger=None)

Fetch and executes all checks configured for the given stage.

Checks are retrieved from the remote deploy host and cached within tmp.

_get_config_overrides()

Get config information locally or from the deployment host.

If the local [repo]-cache/.config file does not exist, or –refresh-config has been passed explicitly on the command line, then the config is fetched from the deployment server. Otherwise the local config cache is used.

This is useful for things like locally rebuilding config files when a remote var file has changed, and the version deployed from the deployment server has changed, but a particular target has not yet been updated.

_get_remote_overrides()

Grab remote config from git_server.

_load_config(use_global_config=True)

Load configuration.

config_deploy()

Render config files.

Grabs the current config yaml file from the deploy git server, and renders the config files templates inside the repo-cache’s .git/config-files directory. During the promote stage, final links are created to the rev-specific config files.

If the config file target path for a template is relative, however, the file is written directly at the target path and no link is created.

config_diff()

Render config files from DEPLOY_HEAD and compare each file to the deployed version. This is called by scap deploy –dry-run

fetch()

Fetch the specified revision of the remote repo.

The given repo is cloned into the cache directory and a new working directory for the given revision is created under revs/{rev}.

At the end of this stage, the .in-progress link is created to signal the possibility for future rollback.

finalize(rollback=False, rev=None)

Perform the final deploy actions.

Moves the .done flag to the rev directory, removes the .in-progress flag, and cleans up old revision directories.

handle_service()

Restart, reload or disable service(s) and check port based on configuration.

main(*extra_args)

Main business logic of the application.

Parsed command line arguments are available in self.arguments. Global configuration is available in self.config. Unparsed command line arguments are passed as positional arguments.

Returns

exit status

promote(rev=None, rev_dir=None, config_deploy=True)

Promote the current deployment.

Switches the current symlink to the current revision directory and restarts the configured service.

Probes the configured service port to measure whether it successfully restarted.

rollback()

Performs a rollback to the last deployed revision.

Rollback looks for a .done symlink that points to the revision directory for the last successful deployment. If this link doesn’t exist, a rollback isn’t possible. If it does exist, the current revision directory is replaced with the target of the link and the promote and finalize stages are re-run.

class scap.DeployLog(exe_name)

Tail/filter/output events from the deploy logs.

examples:

deploy-log -v
deploy-log 'host == scap-target-01'
deploy-log 'msg ~ "some important (message|msg)"'
deploy-log 'levelno >= WARNING host == scap-target-*'

_setup_loggers()

Setup logging.

main(*extra_args)

Main business logic of the application.

Parsed command line arguments are available in self.arguments. Global configuration is available in self.config. Unparsed command line arguments are passed as positional arguments.

Returns

exit status

class scap.DeployPromote(exe_name)

Scap sub-command to promote a specified group of wikis to a specific/latest wmf deployment branch

_commit_files() → bool

Returns True if a commit was created, False if not.

_set_messages()

Craft commit message and scap announcement message

main(*extra_args)

Main business logic of the application.

Parsed command line arguments are available in self.arguments. Global configuration is available in self.config. Unparsed command line arguments are passed as positional arguments.

Returns

exit status

class scap.InstallWorld(exe_name)

Scap sub-command to install scap version on targets

__init__(exe_name)

main(*extra_args)

Main business logic of the application.

Parsed command line arguments are available in self.arguments. Global configuration is available in self.config. Unparsed command line arguments are passed as positional arguments.

Returns

exit status

class scap.LockManager(exe_name)

Holds a lock open for a given repository.

examples:

lock 'Testing something, do not deploy'

main(*extra_args)

Main business logic of the application.

Parsed command line arguments are available in self.arguments. Global configuration is available in self.config. Unparsed command line arguments are passed as positional arguments.

Returns

exit status

class scap.MWVersionsInUse(exe_name)

Get a list of the active MediaWiki versions.

main(*extra_args)

Main business logic of the application.

Parsed command line arguments are available in self.arguments. Global configuration is available in self.config. Unparsed command line arguments are passed as positional arguments.

Returns

exit status

class scap.RebuildCdbs(exe_name)

Rebuild localization cache CDB files from the JSON versions.

main(*extra_args)

Main business logic of the application.

Parsed command line arguments are available in self.arguments. Global configuration is available in self.config. Unparsed command line arguments are passed as positional arguments.

Returns

exit status

class scap.RefreshCdbJsonFiles(exe_name)

Create JSON/MD5 files for all CDB files in a directory.

This will put a JSON and MD5 file in /upstream for each CDB file.

This can be combined with rsync and the scap-rebuild-cdbs to push out large CDB files with minimal traffic. CDB files change drastically with small key/value changes, where as JSON files do not, and thus they diff/rdiff much better.

When pushing updates with rsync, this should be run before running rsync. The rsync command should exclude CDB files or at least use -ignore-existing. After the rsync is done, scap-rebuild-cdbs can be run on each server to apply the updates to the CDB files.

main(*extra_args)

Main business logic of the application.

Parsed command line arguments are available in self.arguments. Global configuration is available in self.config. Unparsed command line arguments are passed as positional arguments.

Returns

exit status

class scap.SecurityPatchCheck(exe_name)

Check if security patches are applied.

class to check if patches in /srv/patches have been applied to the active wikiversions

main(*extra_args)

Main business logic of the application.

Parsed command line arguments are available in self.arguments. Global configuration is available in self.config. Unparsed command line arguments are passed as positional arguments.

Returns

exit status

class scap.StageTrain(exe_name)

Scap sub-command to stage a typical Tuesday deploy of the RelEng train deployment

__init__(exe_name)

main(*extra_args)

Main business logic of the application.

Parsed command line arguments are available in self.arguments. Global configuration is available in self.config. Unparsed command line arguments are passed as positional arguments.

Returns

exit status

class scap.SyncFile(exe_name)

Sync a specific file/directory to the cluster.

main(*extra_args)

Perform a sync operation to the cluster.

class scap.SyncMaster(exe_name)

Sync local MediaWiki staging directory with deploy server state.

main(*extra_args)

Main business logic of the application.

Parsed command line arguments are available in self.arguments. Global configuration is available in self.config. Unparsed command line arguments are passed as positional arguments.

Returns

exit status

class scap.SyncPull(exe_name)

Sync local MediaWiki deployment directory with deploy server state.

main(*extra_args)

Main business logic of the application.

Parsed command line arguments are available in self.arguments. Global configuration is available in self.config. Unparsed command line arguments are passed as positional arguments.

Returns

exit status

class scap.SyncWikiversions(exe_name)

Rebuild and sync wikiversions.php to the cluster.

_before_cluster_sync()

check for the presence of ExtensionMessages and l10n cache for every branch of mediawiki that is referenced in wikiversions.json to avoid syncing a branch that is lacking these critical files.

_update_caches()

Skip this step.

It currently consists only of cache_git_info and this class should attempt to be fast where possible.

class scap.Train(exe_name)

Advance or rollback the train

main(*extra_args)

Main business logic of the application.

Parsed command line arguments are available in self.arguments. Global configuration is available in self.config. Unparsed command line arguments are passed as positional arguments.

Returns

exit status

class scap.Version(exe_name)

main(*extra_args)

Main business logic of the application.

Parsed command line arguments are available in self.arguments. Global configuration is available in self.config. Unparsed command line arguments are passed as positional arguments.

Returns

exit status

scap.cli

Classes and helpers for creating command line interfaces

Copyright © 2014-2017 Wikimedia Foundation and Contributors.

This file is part of Scap.

Scap is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, version 3.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program. If not, see <http://www.gnu.org/licenses/>.

class scap.cli.Application(exe_name)

Base class for creating command line applications.

__init__(exe_name)

_assert_auth_sock()

Assert that SSH_AUTH_SOCK is present in the environment.

_assert_current_user(user)

Assert that this program is run as the given user.

_before_exit(exit_status)

Do any final cleanup or processing before the application exits.

Called after main() and before sys.exit even when an exception occurs.

Returns

exit status

_check_user_auth_sock()

Similar to _assert_auth_sock method above.

Returns immediately when “gerrit_push_user” has been configured, it is assumed the identity is provided in the ssh authentication socket / keyholder.

Unlike _assert_auth_sock, this method:

• Checks the original SSH_AUTH_SOCK provided in the environment by the user, before scap potentially overrides it

• Is meant to be called during operations where a user-provided ssh-agent is required (e.g. deploy-promote). It is not a global requirement of scap

_handle_exception(ex)

Handle unhandled exceptions and errors.

Returns

exit status

_load_config(use_global_config=True)

Load configuration.

_process_arguments(args, extra_args)

Validate and process command line arguments.

Default behavior is to abort the application with an error if any unparsed arguments were found.

Returns

Tuple of (args, extra_args) after processing

_run_as(user)

Ensure that this program is run as the given user.

_setup_environ()

Setup shell environment.

_setup_loggers()

Setup logging.

active_wikiversions(source_tree='deploy', return_type=<class 'list'>)

Get an ordered list or dictionary of active MediaWiki versions.

Parameters

• source_tree – Source tree to read file from: ‘deploy’ or ‘stage’

• return_type – One of list or dict.

Returns

If return_type is list (the default), returns a list of version strings (like “1.38.0-wmf.4”) in ascending order.

If return_type is dict, returns a collections.OrderedDict of {version:wikidb} values sorted by version number in ascending order. ‘wikidb’ will be the first-seen wikidb for ‘version’. This can be used by operations that need a db but don’t care which wiki’s db is used.

announce(*args)

Announce a message to broadcast listeners.

Emits a logging event to the ‘scap.announce’ logger which can be configured to broadcast messages via irc or another real-time notification channel.

Announcements can be disabled by using ‘–no-log-message’ in the command invocation (e.g. scap sync-file –no-log-message foo.php). In this case the log event will still be emitted to the normal logger as though self.get_logger().info() was used instead of self.announce().

format_passthrough_args() → list

Returns a list with user-supplied cli config args in cli format. Makes it easy to pass along config args to locally spawned scap processes.

For example, if the user passed config options canary_dashboard_url and log_json on the command line, then the return value would be:

[“-D”, “canary_dashboard_url:https://somewhere.over.the.rainbow”, “-D”, “log_json:false”]

get_duration()

Get the elapsed duration in seconds.

get_gerrit_ssh_env() → dict

An environment to interact with Gerrit over ssh

When gerrit_push_user scap configuration is set, set the GIT_SSH_COMMAND environment variable to have git ssh to use that user instead of the username of the person that invoked scap.

Else if a user-supplied ssh agent socket was provided, override the one set by scap which does not have the invoking user key.

Eventually we will always require gerrit_push_user to be set.

See https://phabricator.wikimedia.org/T304557

Returns

environment suitable for sshing to Gerrit

get_keyholder_key(ssh_user=None, key_name=None)

Get the private key for IdentityFile use in ssh.

get_lock_file()

Get the path to scap.lock

get_logger()

Lazy getter for a logger instance.

get_master_list()

Get list of deploy master hostnames that should be updated before the rest of the cluster.

get_script_path(remote=False)

Returns the path to the scap script.

handle_keyboard_interrupt()

Handle ctrl-c from interactive user.

Returns

exit status

main(*extra_args)

Main business logic of the application.

Parsed command line arguments are available in self.arguments. Global configuration is available in self.config. Unparsed command line arguments are passed as positional arguments.

Returns

exit status

prompt_for_approval_or_exit(prompt_message, exit_message)

Exits successfully with a message if the user does not approve.

classmethod run()

Construct and run an application.

Calls sys.exit with exit status returned by application by default. Setting exit to False will instead return the class instance and it’s exit status. This would generally only be done when testing.

Parameters

• cls – Class to create and run

• argv – Command line arguments

Returns

Tuple of class instance and exit status when not exiting

scap_check_call(scap_cmd: list)

Call scap subcommand

Raises

CalledProcessError – if the subcommand fails

scap_check_output(scap_cmd: list) → str

Call scap subcommand and return output as text

Returns

the output of scap_cmd as text

Raises

CalledProcessError – if the subcommand fails

scap.cli.all_commands()

return a list of all commands that have been registered with the command() decorator.

@scap.cli.argument(option_flags..[,action='store'][,nargs=1]       [,const=None][,default][,type=str][,choices][,required=False][,help]       [,dest])

Decorator used to declare a command line argument on an Application

Maps a command line argument to the decorated class or method.

Parameters

• option_flags (str) – One or more option flags associated with this argument, e.g. ‘-a’, ‘–arg’

• action – the action associated with this argument. e.g. ‘store_true’

• default – The default value for this argument if not specified by the user.

• help (str) – Short description of this argument, displayed in --help text.

• choices (list) – List possible values for this argument. If specified, argument values will be validated for you and --help will list the possible choices for the user.

• required (bool) – True if your argument is required.

• type (type) – The type of value accepted by your argument, e.g. int or file.

• nargs (int, str) – The number of values accepted by this argument.

@scap.cli.command(command_name, help="help text"[, subcommands=False])

Map a scap sub-command to the decorated class.

Parameters

• command_name (str) – The name of the sub-command

• help (str or None) – A summary of your command to be displayed in –help text

Raises

ValueError – if there is already a command named command_name

Usage Example:

import scap.cli

@cli.command('hello', help='prints "hello world" and exits')
class HelloCommand(cli.Application):
    @cli.argument('--goodbye', action='store_true',
                  help='Say goodbye instead.')
    def main(extra_args):
        if self.arguments.goodbye:
            print('Goodbye, cruel world.')
        else:
            print('Hello, world.')

@scap.cli.subcommand(command_name)

Define an argparse subcommand by decorating a method on your cli.Application subclass.

Use this decorator when you want to have a subcommand on a method other than ‘main’

In order for this to have any affect, your cli.Application must be decorated with subcommands=True (see example below).

Usage Example:

import scap.cli

@cli.command('hello', subcommands=True,
             help='prints "hello world" and exits',)
class HelloCommand(cli.Application):
    @cli.subcommand('world')
    def world_subcommand(extra_args):
        print('hello world')

scap.arg

Helpers for creating a fancy argparser. Most of the externally useful API for command line arg parsing is found in scap.cli

Author

Tyler Cipriani <thcipriani@wikimedia.org>

Author

Mukunda Modell <mmodell@wikimedia.org>

Copyright

Wikimedia Foundation, Inc.

License

GPL v3.0

Parts of the shell argument completion code in this file is derived from python-selfcompletion.

Author

David Barnett <davidbarnett2@gmail.com>

License

BSD

See also

• scap.cli.command()

• scap.cli.argument()

Copyright © 2014-2017 Wikimedia Foundation and Contributors.

This file is part of Scap.

Scap is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, version 3.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program. If not, see <http://www.gnu.org/licenses/>.

class scap.arg.ScapArgParser(*args, **kwargs)

Scap argparse subclass

Created to allow for easier, scripted, autocompletion

__init__(*args, **kwargs)

get_valid_next_words(words, more_valid_words=None)

get completion words for cli auto-complete

class scap.arg.ScapHelpFormatter(prog, indent_increment=2, max_help_position=32, width=None)

Formatter that respects argparse.SUPPRESS for subparser actions.

__init__(prog, indent_increment=2, max_help_position=32, width=None)

class scap.arg._ScapAutoCompleteAction

__call__(parser, namespace, values, option_string=None)

Call self as a function.

__init__()

scap.arg.build_parser()

Build an argument parser for all cli.Application’s.

scap.arg.build_subparser(cmd, parser, global_parser)

Append subparsers to cli.Application’s argparser using decorators.

scap.arg.extract_help_from_object(obj)

Extract help information from the object’s docblock

scap.arg.get_global_parser()

Add standard arguments to argparser.

These arguments should be present on all subparsers.

..info::

The other option with these commands would be to make them into top-level flags for scap; however, that ends up feeling clunky with commands like:

scap –verbose sync

Or

scap -e beta deploy –repo mockbase/deploy

scap.arg.is_dir(string)

represents a cli argument which accepts only a valid directory name

scap.arg.is_version(string)

represents a cli argument which accepts a mediawiki branch version

scap.config

Configuration management

Copyright © 2014-2017 Wikimedia Foundation and Contributors.

This file is part of Scap.

Scap is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, version 3.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program. If not, see <http://www.gnu.org/licenses/>.

scap.config.coerce_value(key, value)

Coerce the given value based on the default config type.

scap.config.load(cfg_file=None, environment=None, overrides=None, use_global_config=True)

Load configuration.

A configuration file consists of sections, led by a [section] header and followed by name: value entries. Lines beginning with '#' are ignored and may be used to provide comments.

A configuration file can contain multiple sections. The configuration object is populated with values from the global section and additional sections based on the fully qualified domain name of the local host. For example, on the host deployXXXX.eqiad.wmnet the final value for a given setting would be the first value found in sections: deployXXXX.eqiad.wmnet, eqiad.wmnet, wmnet or global. Sections not present in the configuration file will be ignored.

Configuration values are loaded from a file specified by the -c or --conf command-line options or from the default locations with the following hierarchy, sorted by override priority:

1. $(pwd)/scap/environments/<environment>/scap.cfg or $(pwd)/scap/scap.cfg (if no environment was specified)

2. /etc/scap.cfg (if use_global_config is true)

For example, if a configuration parameter is set in $(pwd)/scap/scap.cfg and that same parameter is set in /etc/scap.cfg the value for that parameter set in $(pwd)/scap/scap.cfg will be used during execution.

Parameters

• cfg_file – Alternate configuration file

• environment – the string path under which scap.cfg is found

• overrides – Dict of configuration values

• use_global_config – A boolean indicating if /etc/scap.cfg should be read

Returns

dict of configuration values

scap.config.multi_value(str_value)

Given a string that’s got commas, turn it into a list

Parameters

str_value – Random thing the user typed in config

scap.config.override_config(config, overrides=None)

Override values in a config with type-coerced values.

Built-in scap command classes

scap.main

Command wrappers for scap tasks

Copyright © 2014-2017 Wikimedia Foundation and Contributors.

This file is part of Scap.

Scap is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, version 3.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program. If not, see <http://www.gnu.org/licenses/>.

class scap.main.AbstractSync(exe_name)

Base class for applications that want to sync one or more files from the deployment server to the rest of the cluster.

__init__(exe_name)

_apache_sync_command(proxies: list = (), **kwargs) → list

Returns (as a list) the scap pull command to run on mediawiki installation targets. This is comprised of the base scap pull command (defined by _base_scap_pull_command) followed by the list of deployment masters and the list of proxies.

Parameters

• proxies – A list of proxy hostnames that can be pulled from in addition to the deployment masters. Default is empty (as a tuple to prevent mutable list warning)

• kwargs – Any remaining keyword arguments are passed on to _base_scap_pull_command.

_base_scap_pull_command() → list

Returns (as a list) the basic scap pull command to run on a remote target. Note that no source servers are specified in the command so scap pull will default to pull from whatever master_rsync is defined to be in the scap configuration on the target.

_compile_wikiversions()

Compile wikiversions.json to wikiversions.php in stage_dir

_get_api_canary_list()

Get list of MediaWiki api canaries.

_get_app_canary_list()

Get list of MediaWiki api canaries.

_get_canary_list()

Get list of MediaWiki canary hostnames.

_get_proxy_list()

Get list of sync proxy hostnames that should be updated before the rest of the cluster.

_get_target_list()

Get list of hostnames that should be updated from the proxies.

_get_testserver_list()

Get list of MediaWiki testservers.

_git_repo()

Flatten deploy directory into shared git repo.

_master_sync_command()

Synchronization command to run on the master hosts.

_perform_sync(type: str, command: list, targets: list, shuffle=False)

Parameters

• type – A string like “apaches” or “proxies” naming the type of target.

• command – The command to run on the targets, specified as a list.

• targets – A list of strings naming hosts to sync.

• shuffle – If true, the target host list will be randomized.

_restart_php()

If php_fpm_restart_script is set in the configuration then on all dsh groups referenced by the mw_web_clusters config parameter do the following:

Check if php-fpm opcache is full, if so restart php-fpm. If the php_fpm_always_restart config parameter is true, the opcache is treated as always full, so php-fpm will always restart.

If the operator invoked scap with the –force flag, restart php-fpm unsafely (i.e., without depooling and repooling around the service restart). T243009

Targets that have already been restarted will not be restarted again.

_restart_php_hostgroups(target_hosts=None)

Perform php restart for sets of hosts (if configured).

Parameter target_hosts is a list of lists of hostnames.

_setup_php()

Sets up the php_fpm instance if not already initialized.

Returns True if php_fpm restart has been configured, or False if not.

_sync_masters()

Sync the staging directory across all other deploy master servers.

canary_checks()

Run logstash error rate checks (for bare metal and mw-on-k8s).

Raises

SystemExit – on canary check failure

check_testservers(baremetal_testservers: list)

Check bare metal and k8s testservers.

Raises

SystemExit – on check failure

get_keyholder_key()

Returns scap2-specific deploy key

This way we can set a key in the default scap config without having all non-scap2 repos inherit that configuration.

increment_stat(stat, all_stat=True, value=1)

Increment a stat in deploy.*

Parameters

• stat – String name of stat to increment

• all_stat – Whether to increment deploy.all as well

• value – How many to increment by, default of 1 is normal

main(*extra_args)

Perform a sync operation to the cluster.

master_only_cmd(timer, cmd)

Run a command on all other master servers than the one we’re on

Parameters

• timer – String name to use in timer/logging

• cmd – List of command/parameters to be executed

retry_continue_exit(description, test_func)

Runs test_func(). If it returns True, this function returns. If test_func() does not return true:

• If an interactive terminal is not available, self.cancel() is called and it is not expected to return.

• If an interactive terminal is available, ask the user if they want to retry the test, continue with deployment anyway, or exit. If the user chooses to retry, this function starts over. If the user chooses to continue, a message is logged and this function returns. If the user chooses to exit, self.cancel() is called and it is not expected to return.

‘description’ is used to describe the operation to retry in the retry/continue/exit prompt.

sync_targets(targets=None, type=None)

This function is used to sync to bare metal testservers and canaries.

Run scap pull on the targets, including l10n rebuild, wikiversions sync, and php-fpm restart. The pull source will be this deploy server.

Parameters

• targets – Iterable of target servers to sync

• type – A string like “testservers” or “canaries” naming the type of target.

class scap.main.CompileWikiversions(exe_name)

Compile wikiversions.json to wikiversions.php.

main(*extra_args)

Main business logic of the application.

Parsed command line arguments are available in self.arguments. Global configuration is available in self.config. Unparsed command line arguments are passed as positional arguments.

Returns

exit status

class scap.main.LockManager(exe_name)

Holds a lock open for a given repository.

examples:

lock 'Testing something, do not deploy'

main(*extra_args)

Main business logic of the application.

Parsed command line arguments are available in self.arguments. Global configuration is available in self.config. Unparsed command line arguments are passed as positional arguments.

Returns

exit status

class scap.main.MWVersionsInUse(exe_name)

Get a list of the active MediaWiki versions.

main(*extra_args)

Main business logic of the application.

Parsed command line arguments are available in self.arguments. Global configuration is available in self.config. Unparsed command line arguments are passed as positional arguments.

Returns

exit status

class scap.main.RebuildCdbs(exe_name)

Rebuild localization cache CDB files from the JSON versions.

main(*extra_args)

Main business logic of the application.

Parsed command line arguments are available in self.arguments. Global configuration is available in self.config. Unparsed command line arguments are passed as positional arguments.

Returns

exit status

class scap.main.RefreshCdbJsonFiles(exe_name)

Create JSON/MD5 files for all CDB files in a directory.

This will put a JSON and MD5 file in /upstream for each CDB file.

This can be combined with rsync and the scap-rebuild-cdbs to push out large CDB files with minimal traffic. CDB files change drastically with small key/value changes, where as JSON files do not, and thus they diff/rdiff much better.

When pushing updates with rsync, this should be run before running rsync. The rsync command should exclude CDB files or at least use -ignore-existing. After the rsync is done, scap-rebuild-cdbs can be run on each server to apply the updates to the CDB files.

main(*extra_args)

Main business logic of the application.

Parsed command line arguments are available in self.arguments. Global configuration is available in self.config. Unparsed command line arguments are passed as positional arguments.

Returns

exit status

class scap.main.ScapWorld(*args, **kwargs)

Deploy MediaWiki to the cluster.

1. Validate php syntax of wmf-config and multiversion

2. Compile wikiversions.json to php in staging directory

3. Update l10n files in staging area

4. Compute git version information

5. Ask scap masters to sync with current master

6. Ask scap proxies to sync with master server

7. Ask apaches to sync with fastest rsync server (excluding wikiversions.php)

8. Ask apaches to rebuild l10n CDB files

9. Ask apaches to sync wikiversions.php

10. Run purgeMessageBlobStore.php

11. Rolling invalidation of all opcache for php 7.x

__init__(*args, **kwargs)

_before_exit(exit_status)

Do any final cleanup or processing before the application exits.

Called after main() and before sys.exit even when an exception occurs.

Returns

exit status

_handle_exception(ex)

Handle unhandled exceptions and errors.

Returns

exit status

main(*extra_args)

Perform a sync operation to the cluster.

class scap.main.SecurityPatchCheck(exe_name)

Check if security patches are applied.

class to check if patches in /srv/patches have been applied to the active wikiversions

main(*extra_args)

Main business logic of the application.

Parsed command line arguments are available in self.arguments. Global configuration is available in self.config. Unparsed command line arguments are passed as positional arguments.

Returns

exit status

class scap.main.SyncFile(exe_name)

Sync a specific file/directory to the cluster.

main(*extra_args)

Perform a sync operation to the cluster.

class scap.main.SyncMaster(exe_name)

Sync local MediaWiki staging directory with deploy server state.

main(*extra_args)

Main business logic of the application.

Parsed command line arguments are available in self.arguments. Global configuration is available in self.config. Unparsed command line arguments are passed as positional arguments.

Returns

exit status

class scap.main.SyncPull(exe_name)

Sync local MediaWiki deployment directory with deploy server state.

main(*extra_args)

Main business logic of the application.

Parsed command line arguments are available in self.arguments. Global configuration is available in self.config. Unparsed command line arguments are passed as positional arguments.

Returns

exit status

class scap.main.SyncWikiversions(exe_name)

Rebuild and sync wikiversions.php to the cluster.

_before_cluster_sync()

check for the presence of ExtensionMessages and l10n cache for every branch of mediawiki that is referenced in wikiversions.json to avoid syncing a branch that is lacking these critical files.

_update_caches()

Skip this step.

It currently consists only of cache_git_info and this class should attempt to be fast where possible.

class scap.main.Version(exe_name)

main(*extra_args)

Main business logic of the application.

Parsed command line arguments are available in self.arguments. Global configuration is available in self.config. Unparsed command line arguments are passed as positional arguments.

Returns

exit status

Plug-in scap sub-command classes

scap.plugins

Scap plugin architecture

scap.plugins.find_plugins(plugin_dirs)

Get a list of all plugins found in in plugin_dirs

Parameters

plugin_dirs (list) – directories to search for plugins

Returns

list of all plugin commands found in plugin_dirs

scap.plugins.load_plugins([plugin_dir])

load scap plugin modules.

Parameters

plugin_dir (str) – an additional location to search

Copyright © 2014-2017 Wikimedia Foundation and Contributors.

This file is part of Scap.

Scap is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, version 3.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program. If not, see <http://www.gnu.org/licenses/>.

class scap.plugins.Say(exe_name)

Scap propaganda of the lowest order.

main(*extra_args)

Main business logic of the application.

Parsed command line arguments are available in self.arguments. Global configuration is available in self.config. Unparsed command line arguments are passed as positional arguments.

Returns

exit status

See also

Writing scap plugins

Command Execution

scap.cmd

command execution

Copyright © 2014-2017 Wikimedia Foundation and Contributors.

This file is part of Scap.

Scap is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, version 3.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program. If not, see <http://www.gnu.org/licenses/>.

class scap.cmd.Command(*cmds)

Command is used to store a parameterized shell command for reuse by cluster_ssh and other utilities. The idea is that we store the command, along with any arguments that need to be filled in, then we can call the command later just like a normal python function, which returns the shell command to be ran.

Usage Example:

import scap.cmd as cmd

ssh = cmd.Command('/usr/bin/ssh', cmd.arg('user', '-oUser={}'))
sudo = cmd.Command('sudo', cmd.arg('user', '-u {}'), '-n', '--')
ssh('some.host', sudo('remote_cmd', 'some', 'args', user='sudo_user'),
    user='ssh_user')

# result:

['/usr/bin/ssh',
 '-oUser=ssh_user',
 'some.host',
 'sudo',
 '-u sudo_user',
 '-n',
 '--',
 'remote_cmd',
 'some',
 'args']

__call__(*args, **values)

Call self as a function.

__init__(*cmds)

class scap.cmd.arg(name, cmd)

Represent a named parameter that will be passed to an instance of Command.

For example, arg('user', '-u {}') creates an arg called ‘user’ which will evaluate to "-u luser" when the command is evaluated with some_command(user='luser')

Parameters

• name – The name of the parameter, used to specify it’s value later when the command is evaluated.

• cmd – string, the argument text format string.

__call__(**values)

Call self as a function.

__init__(name, cmd)

required(required=True)

Mark this argument as required

scap.ssh

This module provides functions for running commands on remote hosts via SSH.

Copyright © 2014-2017 Wikimedia Foundation and Contributors.

This file is part of Scap.

Scap is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, version 3.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program. If not, see <http://www.gnu.org/licenses/>.

class scap.ssh.JSONOutputHandler(host)

Deserialize and log structured JSON output from hosts.

Any non-structured output is stored for future handling.

__init__(host)

accept(output)

Extract and deserializes line-wise JSON from the given output.

Any non-JSON is stored in self.output.

lines(output)

Generate each line of the given output.

Reconstructs partial lines using the leftovers from previous calls.

class scap.ssh.Job(hosts=None, command=None, user=None, logger=None, key=None, verbose=False)

Execute a job on a group of remote hosts via ssh.

__init__(hosts=None, command=None, user=None, logger=None, key=None, verbose=False)

command(command)

Set command to run.

get_logger()

Lazy getter for a logger instance.

hosts(hosts)

Set hosts to run command on.

progress(reporter)

Set the reporter used when reporting progress.

run(batch_size=80, return_jobresults=False)

Run the job, report progress, and return success/failed counts or a JobResults object if return_jobresults is truthy.

Returns

(ok, failed) counts of successful/failed hosts or a JobResults object if return_jobresults is truthy.

Raises

RuntimeError if command has not been set

run_with_status(batch_size=80)

Run the job, report progress, and yield JobResult objects as execution completes.

Yields

JobResult

Raises

RuntimeError if command has not been set

shuffle()

Randomize order of target hosts.

class scap.ssh.OutputHandler(host)

Standard handler for SSH command output from hosts.

Simply stores output as a string for future handling.

__init__(host)

scap.ssh.cluster_ssh(hosts, command, user=None, key=None, limit=80, max_fail=None, output_handler=None, verbose=False, reporter=None)

Run a command via SSH on multiple hosts concurrently.

scap.tasks

Contains functions implementing scap tasks

Copyright © 2014-2017 Wikimedia Foundation and Contributors.

This file is part of Scap.

Scap is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, version 3.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program. If not, see <http://www.gnu.org/licenses/>.

scap.tasks._call_rebuildLocalisationCache(version, out_dir, use_cores=1, php_l10n=False, lang=None, force=False, quiet=False, delay_messageblobstore_purge=False)

Helper for update_localization_cache.

Parameters

• version – The train version to build l10n for

• out_dir – The output directory

• use_cores – The number of cores to run in

• lang – The –lang option, or None to omit

• force – Whether to pass –force

• quiet – Whether to pass –quiet

scap.tasks.cache_git_info(version, cfg)

Create JSON cache files of git branch information.

Parameters

• version – MediaWiki version (eg ‘1.38.0-wmf.20’)

• cfg – Dict of global configuration values

Raises

IOError if version directory is not found

scap.tasks.check_patch_files(version, cfg)

Check to see if there are unmerged patch files from /srv/patches for a given revision.

Parameters

• version – MediaWiki version string (e.g., ‘1.27.0-wmf.8’)

• cfg – Scap configuration dict

scap.tasks.clear_message_blobs(cfg, logger=None)

Clear MessageBlobStore cache on all wikis

Parameters

• cfg – Scap configuration dict

• logger – logger instance

scap.tasks.compile_wikiversions(source_tree, cfg, logger=None)

Validate and compile the wikiversions.json file.

1. Find the realm specific filename for wikiversions.json in specified tree (deploy or staging)

2. Validate that all versions mentioned in the json exist as directories in the specified tree.

3. Validate that all wikis in the json file are members of (realm-specific) dblists/all.dblist.

4. Create a temporary php file from the json contents

5. Atomically rename the temporary php to the realm specific wikiversions.php filename

Parameters

• source_tree – Source tree to read file from: ‘deploy’ or ‘stage’

• cfg – Dict of global configuration values

scap.tasks.ensure_extension_messages_file(cfg, version, logger)

Ensure that <staging>/wmf-config/ExtensionMessages-<version>.php exists. If it does not exist, create a blank file.

In either case, the path to the ExtensionMessages-<version>.php file is returned.

scap.tasks.get_old_wikiversions(versions, keep=2, keep_static=5)

Get lists of old MediaWiki versions to be removed

Parameters

• keep=2 – Number of branches for which we want to keep everything

• keep_static=5 – Number of weeks to keep static assets

Returns

tuple of lists of old wikiversions

scap.tasks.get_wikiversions_ondisk(directory) → list

Returns a list of the train branch versions that are currently checked out in DIRECTORY. The list is sorted in ascending order.

Returns

list like:: [‘1.41.0-wmf.1’, ‘1.41.0-wmf.2’]

scap.tasks.get_wikiversions_ondisk_ex(directory)

Get checked-out wikiversions in a directory.

Finds wikiversions in a directory and does its best to determine the date of the oldest reflog for that branch (non recursive)

Returns

list of tuples like:: [(“1.29.0-wmf.17”, <DateCreated>)]`

scap.tasks.handle_services(services, require_valid_service=False, on_secondary_host=False)

Take a comma-separated list of services and restart, reload or disable each of them.

The idea is to take a string directly from the scap.cfg file that looks like:

jobrunner, jobchron=reload, jenkins=restart:disable-secondary, …

and be able to determine what to do with that list.

If no action is specified the default is “restart”. When specified, main action can be one of:

• restart: service will be restarted

• reload: service will be reloaded

When specified, secondary action can be one of:

• disable-secondary: service is disabled on targets marked as secondary

Note that specifying a secondary action requires a main action

scap.tasks.logstash_canary_checks(service, threshold, logstash, delay)

Run the logstash canary checks on test application servers.

Parameters

• threshold – float, average log multiple at which to fail

• service – string, name of the service to check

• logstash – string, logstash server

• delay – float, time between deploy and now

Returns the exit status of logstash_checker.py

scap.tasks.merge_cdb_updates(directory, pool_size, trust_mtime=False, mute=False, logger=None)

Update l10n CDB files using JSON data.

Parameters

• directory – L10n cache directory

• pool_size – Number of parallel processes to use

• trust_mtime – Trust file modification time?

• mute – Disable progress indicator

scap.tasks.refresh_cdb_json_file(cdb_file_path) → bool

Rebuild json file from cdb file.

1. Check md5 file saved in upstream against md5 of cdb file

2. Read cdb file to dict

3. Write dict to named temporary file

4. Change permissions on named temporary file

5. Overwrite upstream json file

6. Write upstream md5 file

Returns a boolean indicating whether or not a JSON file was (re)created.

scap.tasks.refresh_cdb_json_files(in_dir, pool_size, verbose)

Update json files from corresponding cdb file in parallel.

Parameters

• in_dir – directory containing cdb files

• pool_size – number of “threads” to use

• verbose – output verbosely

scap.tasks.sync_common(cfg, include=None, sync_from=None, verbose=False, logger=None, rsync_args=None, exclude_wikiversionsphp=False)

Sync local deploy dir with upstream rsync server’s copy.

Rsync from server::common to the local deploy directory. If a list of servers is given in sync_from we will attempt to select the “best” one to sync from. If no servers are given or all servers given have issues we will fall back to using the server named by master_rsync in the configuration data.

Parameters

• cfg – Dict of global configuration values.

• include – List of rsync include patterns to limit the sync to. If None is given the entire common module on the target rsync server will be transferred. Rsync syntax for syncing a directory is <dirname>/***.

• sync_from – List of rsync servers to fetch from.

scap.tasks.sync_master(cfg, master, verbose=False, logger=None)

Sync local staging dir with upstream rsync server’s copy.

Rsync from server::common to the local staging directory.

Parameters

• cfg – Dict of global configuration values.

• master – Master server to sync with

• verbose – Enable verbose logging?

scap.tasks.sync_wikiversions(hosts, cfg, key=None)

Rebuild and sync wikiversions.php to the cluster.

Parameters

• hosts – List of hosts to sync to

• cfg – Dict of global configuration values

scap.tasks.update_l10n_cdb(cache_dir, cdb_file, trust_mtime=False, logger=None)

Update a localization CDB database.

Parameters

• cache_dir – L10n cache directory

• cdb_file – L10n CDB database

• trust_mtime – Trust file modification time?

scap.tasks.update_l10n_cdb_wrapper(args, logger=None)

Wrapper for update_l10n_cdb to be used in contexts where only a single argument can be provided.

Parameters

args – Sequence of arguments to pass to update_l10n_cdb

scap.tasks.update_localization_cache(version, app, logger=None)

Update the localization cache for a given MW version.

Parameters

• version – MediaWiki version

• wikidb – Wiki running given version

• app – Application calling the function

Git Repository Deployment

scap.deploy

Command wrappers for deploy tasks

Copyright © 2014-2017 Wikimedia Foundation and Contributors.

This file is part of Scap.

Scap is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, version 3.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program. If not, see <http://www.gnu.org/licenses/>.

class scap.deploy.Deploy(exe_name)

Sync new service code across cluster.

__init__(exe_name)

_build_deploy_groups()

Build server groups based on configuration server_groups variable

_execute_for_group(stages, group, ignore_failure=False, prompt_user=False)

Executes the given stages across targets of the given group’s subgroups.

Parameters

• stages – List of stages to execute

• group – The targets.DeployGroup for which to execute the stages. Any target host that is unreachable via SSH will be added to the group’s list of excluded hosts.

• ignore_failure – Whether to keep on rolling past the failure_limit threshold. Note that even with this argument, SSH failure result in the target being excluded from future stage execution.

• prompt_user – Whether to prompt the user after each subgroup.

_get_stage_name(stage)

Map a stage name to a stage display name.

_load_config(use_global_config=True)

Set the host directory after the config has been loaded.

_needs_latest_sha1(stages)

Determine whether we expect a new SHA1 to be tagged and deployed

Stages

list of stages being run

_setup_loggers()

Set up additional logging to scap/deploy.log.

checks_setup()

Build info to run checks.

config_deploy_setup(commit)

Generate environment-specific config file and variable template list.

Builds a yaml file that contains: 1. A list of file objects containing template files to be deployed 2. An object containing variables specified in the environment-specific vars.yaml file and inheriting from the vars.yaml file

execute_stage_on_group(stage, group, targets)

Execute a deploy stage for the given group targets.

Parameters

• stage – deploy stage.

• group – deploy group.

• targets – group targets.

Yields

(host, status)

main(*extra_args)

Main business logic of the application.

Parsed command line arguments are available in self.arguments. Global configuration is available in self.config. Unparsed command line arguments are passed as positional arguments.

Returns

exit status

exception scap.deploy.DeployGroupFailure

Signal that a particular deploy group failed

class scap.deploy.DeployLocal(exe_name)

Command that runs on target hosts.

Responsible for fetching code from the git server, checking out the appropriate revisions, restarting/disabling services and running checks.

__init__(exe_name)

static _check_applies(chk, stage, group, when)

Verify whether a check applies for the current group and stage.

_execute_checks(when, stage, group=None, logger=None)

Fetch and executes all checks configured for the given stage.

Checks are retrieved from the remote deploy host and cached within tmp.

_get_config_overrides()

Get config information locally or from the deployment host.

If the local [repo]-cache/.config file does not exist, or –refresh-config has been passed explicitly on the command line, then the config is fetched from the deployment server. Otherwise the local config cache is used.

This is useful for things like locally rebuilding config files when a remote var file has changed, and the version deployed from the deployment server has changed, but a particular target has not yet been updated.

_get_remote_overrides()

Grab remote config from git_server.

_load_config(use_global_config=True)

Load configuration.

config_deploy()

Render config files.

Grabs the current config yaml file from the deploy git server, and renders the config files templates inside the repo-cache’s .git/config-files directory. During the promote stage, final links are created to the rev-specific config files.

If the config file target path for a template is relative, however, the file is written directly at the target path and no link is created.

config_diff()

Render config files from DEPLOY_HEAD and compare each file to the deployed version. This is called by scap deploy –dry-run

fetch()

Fetch the specified revision of the remote repo.

The given repo is cloned into the cache directory and a new working directory for the given revision is created under revs/{rev}.

At the end of this stage, the .in-progress link is created to signal the possibility for future rollback.

finalize(rollback=False, rev=None)

Perform the final deploy actions.

Moves the .done flag to the rev directory, removes the .in-progress flag, and cleans up old revision directories.

handle_service()

Restart, reload or disable service(s) and check port based on configuration.

main(*extra_args)

Main business logic of the application.

Parsed command line arguments are available in self.arguments. Global configuration is available in self.config. Unparsed command line arguments are passed as positional arguments.

Returns

exit status

promote(rev=None, rev_dir=None, config_deploy=True)

Promote the current deployment.

Switches the current symlink to the current revision directory and restarts the configured service.

Probes the configured service port to measure whether it successfully restarted.

rollback()

Performs a rollback to the last deployed revision.

Rollback looks for a .done symlink that points to the revision directory for the last successful deployment. If this link doesn’t exist, a rollback isn’t possible. If it does exist, the current revision directory is replaced with the target of the link and the promote and finalize stages are re-run.

class scap.deploy.DeployLog(exe_name)

Tail/filter/output events from the deploy logs.

examples:

deploy-log -v
deploy-log 'host == scap-target-01'
deploy-log 'msg ~ "some important (message|msg)"'
deploy-log 'levelno >= WARNING host == scap-target-*'

_setup_loggers()

Setup logging.

main(*extra_args)

Main business logic of the application.

Parsed command line arguments are available in self.arguments. Global configuration is available in self.config. Unparsed command line arguments are passed as positional arguments.

Returns

exit status

class scap.deploy.DeployMediaWiki(exe_name)

Deploy mediawiki via scap3.

Ideally, this class and command will be fully merged with the scap deploy command by the end of the transition; however, there may also be unique reasons, particularly on the deployment masters, to keep this command

main(*extra_args)

Run deploy-mediawiki.

scap.git

Helpers for git operations and interacting with .git directories

scap.git.add_all(location, message='Update')

Add everything to repo at location as user.

scap.git.checkout(location, rev)

Checkout a git repo sha at a location

scap.git.clean_tags(location, max_tags)

Make sure there aren’t more than max_tags.

scap.git.clone_or_update_repo(dir, repo, branch, logger, reference=None, ref=None)

Clone or update the checkout of ‘repo’ in ‘dir’, using the specified branch. Note that existing repos are hard-reset to the match the state of the origin.

Submodules are handled as well.

Returns the sha1 of the checked out branch.

scap.git.default_ignore(location)

Create a default .gitignore file.

scap.git.describe(location)

Returns a convenient label for the current state of the git repo.

scap.git.ensure_dir(location)

Ensure that we’re in a git directory. If not, explode

scap.git.fat_init(location)

Initializes the given directory for git-fat use.

scap.git.fat_isinitialized(location)

Returns whether git-fat has been initialized for the given directory.

scap.git.fetch(location, repo, reference=None, dissociate=True, recurse_submodules=False, shallow=False, bare=False, config=None, branch=None)

Fetch a git repo to a location

scap.git.file_has_unstaged_changes(file, location=None) → bool

Untracked files are also considered

scap.git.garbage_collect(location)

Clean up a repo.

scap.git.info(directory, remote='origin')

Compute git version information for a given directory that is compatible with MediaWiki’s GitInfo class.

Parameters

directory – Directory to scan for git information

Returns

Dict of information about current repository state

scap.git.info_filename(directory, install_path, cache_path)

Compute the path for a git_info cache file related to a given directory.

>>> info_filename('foo', 'foo', '')
'info.json'
>>> info_filename('foo/bar/baz', 'foo', 'xyzzy')
'xyzzy/info-bar-baz.json'

scap.git.is_dir(path)

Checks if path is a git, doesn’t count submodule directories

scap.git.largefile_pull(location, implementor, submodules=False)

Syncs all git-fat or git-lfs objects for the given repo directory.

Parameters

• location – Repository to work in

• implementor – What implementation to pull with (git-lfs, git-fat)

scap.git.last_deploy_tag(location)

Finds the last tag to use for this deployment

scap.git.lfs_install(*args)

Run git-lfs-install with provided arguments.

scap.git.list_submodules_paths_urls(repo, args)

Return a list of the paths and URLs of the submodules of the given repository, separated by a space

scap.git.next_deploy_tag(location)

Calculates the scap/sync/YYYY-MM-DD/{n} tag to use for this deployment

scap.git.parse_submodules(location) → dict

Reads .gitmodules and returns a dictionary of information about each defined submodule.

The key is the name of the submodule, and the value is a dictionary with “path” (typically the same as the submodule name) and “url” keys.

scap.git.reflog(repo, fmt='oneline', branch=None)

Fetch reflog as list

scap.git.remap_submodules(location, server)

Remap all submodules to deployment server

This function supports remapping submodules available on the deployment server. Since the remote is a non-bare repo (as of git 1.7.8) all submodules should be available over http under the remote server checkout’s git dir: [server]/[repo]/.git/modules/[submodule_path]

Parameters

• location – String path to local git checkout containing a .gitmodules file

• server – String path to remote, non-bare, repo gitdir

scap.git.remote_exists(location, remote)

Check if remote exists in location

scap.git.remote_set(location, repo, remote='origin')

set the remote at location to repo

scap.git.remove_all_ignores(location)

Remove .gitignore files under a location.

scap.git.resolve_gitdir(directory)

Find the .git directory for a given path.

This will resolve the gitlink if path/.git is a gitlink to a bare repo e.g. a file with one line, like this:

gitdir:/path/to/somewhere/else

scap.git.sha(location, rev)

Returns SHA1 for things like HEAD or HEAD~~

scap.git.sync_submodules(location)

Sync git submodules on target machines

scap.git.tag_repo(deploy_info, location='/srv/app')

creates new tag in deploy repo

scap.git.update_deploy_head(deploy_info, location)

updates .git/DEPLOY_HEAD file

Parameters

• deploy_info – current deploy info to write to file as YAML

• location ((optional)) – git directory location (default cwd)

scap.git.update_server_info(has_submodules=False, location='/srv/app', logger=None)

runs git update-server-info and tags submodules

scap.git.update_submodules(location, git_remote=None, use_upstream=False, reference=None, checkout=False, force=False, lfs_smudge=False)

Update git submodules on target machines

scap.context

Management of deployment host/target directories and execution context.

Copyright © 2014-2017 Wikimedia Foundation and Contributors.

This file is part of Scap.

Scap is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, version 3.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program. If not, see <http://www.gnu.org/licenses/>.

class scap.context.Context(root, environment=None)

Base context for either the deployment host or target.

__init__(root, environment=None)

Instantiate a new context at the given root path.

Parameters

• root – Root directory

• environment – Environment name used when resolving config files.

path(*relpaths)

Qualify path relative to the root path.

setup()

Create the root directory, use it as the root context.

class scap.context.HostContext(root, environment=None)

Manage deployment host paths and execution context.

env_specific_path(*relpaths)

Return path to default or environment specific file/directory.

Both the environment specific path at scap/environments/{name} and the default path at scap is searched in respective order. The first path that exists will be returned.

env_specific_paths(*relpaths)

Return paths to default and environment specific files/directories.

Both the environment specific path at scap/environments/{name} and the default path at scap is searched. Paths are included in the returned list for each one that exists.

log_path(*relpaths)

Qualify the given log path.

scap_path(*relpaths)

Qualify path relative to the scap directory.

setup()

Create the scap and log directories as necessary.

See :class:Context.setup for its additional operations.

class scap.context.TargetContext(root, environment=None)

Manage target host paths and execution context.

property cache_dir

Path to the cached repo clone.

property current_link

Symlink that points to the currently deployed revision.

property current_rev_dir

Real path to the currently deployed revision.

property done_rev_dir

Real path to the revision previously marked as done.

find_old_rev_dirs(cache_revs=5)

Generate revision directories that are candidates for deletion.

The cache_revs most recent revision directories and any revision directory that is current or in progress is not considered.

link_path_to_rev(path, rev, backup=False)

Create link to the given rev’s directory at the given path.

property local_config

Local target file that has a copy of the last-deployed config.

mark_rev_current(rev)

Change the current rev to the given one.

This state is maintained as a current symlink in the directory root that points to the relevant revs/{rev} directory.

mark_rev_done(rev)

Change the state to done for the given rev.

This state is maintained as a .done symlink in the directory root that points to the relevant revs/{rev} directory.

mark_rev_in_progress(rev)

Change the state to in-progress for the given rev.

This state is maintained as a .in-progress symlink in the directory root that points to the relevant revs/{rev} directory.

property rev_done

The rev that is currently marked as done.

property rev_in_progress

The rev that is currently marked as in-progress.

rev_path(rev, *paths)

Return the path to the given repo revision.

property revs_dir

Context directory that stores revisions.

scripts_dir(rev)

Path to scripts for a given rev.

setup()

Create the cache and revs directory.

See :class:Context.setup for its additional operations.

General Utilities

scap.utils

Contains misc utility functions.

class scap.utils.VarDumpJSONEncoder(*, skipkeys=False, ensure_ascii=True, check_circular=True, allow_nan=True, sort_keys=False, indent=None, separators=None, default=None)

encode python objects to json

default(o)

Implement this method in a subclass such that it returns a serializable object for o, or calls the base implementation (to raise a TypeError).

For example, to support arbitrary iterators, you could implement default like this:

def default(self, o):
    try:
        iterable = iter(o)
    except TypeError:
        pass
    else:
        return list(iterable)
    # Let the base class default method raise the TypeError
    return JSONEncoder.default(self, o)

scap.utils._listfiles(dirname)

Generate each pathname for each regular file under dirname

scap.utils.cd(dirname)

Context manager. Cds to dirname.

It moves back to previous dir on context exit. :param dirname: directory into which it should change

scap.utils.context_logger(context_name, *args)

Context manager that maintains nested logger contexts.

Each time you enter a with block using this context manager, a named logger is set up as a child of the current logger. When exiting the with block, the logger gets popped off the stack and the parent logger takes it’s place as the ‘current’ logging context.

The easiest way to use this is to decorate a function with log_context, For Example:

@log_context('name')
def my_func(some, args, logger=None):
    logger.debug('something')

scap.utils.cpus_for_jobs()

Get how many CPUs we can use for farming jobs out

scap.utils.dir_is_empty(path)

Returns True if ‘path’ is an empty directory, otherwise returns False. Raises an error if ‘path’ does not name a directory.

scap.utils.eintr_retry(func, *args)

Retry a system call if it is interrupted by EINTR.

Extracted from stdlib’s subprocess (where it is called _eintr_retry_call – the leading underscore indicating it is not part of the module’s API). This is not needed on Python >= 3.5, thanks to PEP 0475.

See <https://www.python.org/dev/peps/pep-0475/>.

scap.utils.find_nearest_host(hosts, port=22, timeout=1)

Given a collection of hosts, find the one that is the fewest number of hops away.

>>> # Begin test fixture
>>> import socket
>>> fixture_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
>>> fixture_socket.bind(('127.0.0.1', 0))
>>> fixture_socket.listen(1)
>>> fixture_port = fixture_socket.getsockname()[1]
>>> # End test fixture
>>> find_nearest_host(['127.0.0.1'], port=fixture_port)
'127.0.0.1'

Parameters

• hosts – Hosts to check

• port – Port to try to connect on (default: 22)

• timeout – Timeout in seconds (default: 1)

scap.utils.find_regular_files(dirname)

Return sorted list of all regular files under a directory

scap.utils.find_upwards(name, starting_point='/srv/app')

Search the specified directory, and all parent directories, for a given filename, returning the first matching path that is found.

Parameters

• name – the relative path name to search for

• starting_point – the directory to start searching from. Each parent directory will be searched until a match is found.

Returns

if a match is found, returns the full path to the matching file, None otherwise.

scap.utils.get_active_wikiversions(directory, realm, return_type=<class 'list'>)

Get an ordered collection of active MediaWiki versions.

Returns

If ‘return_type’ is list (the default), returns a list of versions, sorted in ascending order.

If ‘return_type’ is dict (the default), returns a collections.OrderedDict of {version:wikidb} values sorted by version number in ascending order. ‘wikidb’ will be the first-seen wikidb for ‘version’. This can be used by operations that need a db but don’t care which wiki’s db is used.

scap.utils.get_current_train_info(api_url, proxy=None) → dict

Returns a dictionary containing information about this week’s train

scap.utils.get_current_train_version_from_gerrit(gerrit_url) → str

Returns a string like ‘1.39.0-wmf.19’

scap.utils.get_env_specific_filename(path, env=None)

Find a file specific to the environment in which scap is running.

scap.utils.get_group_versions(group, directory, realm) → list

Returns a list of versions used by ‘group’, in ascending version order.

scap.utils.get_patches(sub_dirs, root_dir)

Find all patches under each subdirectory.

:param sub_dirs list of sub directories under which to search :param root_dir base path under which subdirectories reside :return dictionary of patches, keyed by sub_dir

scap.utils.get_real_user_fullname()

Return the first entry in GECOS field for real user.

scap.utils.get_real_username()

Get the username of the real user.

scap.utils.get_realm_specific_filename(filename, realm)

If a realm-specific version of ‘filename’ exists, return it, otherwise return ‘filename’. To construct the realm-specific filename, “-REALM” is inserted before the file extension. For example, “wikiversions.json” becomes “wikiversions-REALM.json”.

scap.utils.get_train_blockers_info(api_url, proxy=None) → dict

Returns a dictionary with details about the current and upcoming train blocker tasks

scap.utils.get_user_fullname(name=None)

Return the first entry in GECOS field for name.

scap.utils.get_username(user=None)

Get the username of the effective user.

scap.utils.human_duration(elapsed)

Format an elapsed seconds count as human readable duration.

>>> human_duration(1)
'00m 01s'
>>> human_duration(65)
'01m 05s'
>>> human_duration(60*30+11)
'30m 11s'

scap.utils.is_phabricator_task_id(string: str) → bool

Returns true if ‘string’ has the format of a phabricator task id

scap.utils.isclose(a, b, rel_tol=1e-09, abs_tol=0.0)

Return True if a is close in value to b. False otherwise.

Parameters

• a – one of the values to be tested

• b – the other value to be tested

• rel_tol=1e-9 – The relative tolerance – the amount of error allowed, relative to the absolute value of the larger input values.

• abs_tol=0.0 – The minimum absolute tolerance level – useful for comparisons to zero.

Copyright: Christopher H. Barker Original License: Apache License 2.0 <https://github.com/PythonCHB/close_pep>

scap.utils.iterate_subdirectories(root)

Generator over the child directories of a given directory.

scap.utils.join_path(*fragments)

Join several path fragments into a complete, normalized path string.

Strips leading and trailing slashes from path fragments to avoid an unfortunate feature of os.path.join() which is described in the python documentation for os.path as follows:

“If any component is an absolute path, all previous components are thrown away, and joining continues.”

• https://docs.python.org/2/library/os.path.html#os.path.join

scap.utils.list_intersection(list1, list2)

Returns a list containing the intersection (items in common) of list1 and list2

scap.utils.list_union(list1, list2)

Returns a list containing the union of list1 and list2

scap.utils.log_context(context_name)

Decorator to wrap the a function in a new context_logger.

The logger is passed to the function via a kwarg named ‘logger’.

scap.utils.make_sudo_check_call_env(env)

Returns a string of environment variables formatted for the shell

sudo 1.8.21 adds support for adding a list of variables to –preserve-env, that should replace this function in future

scap.utils.md5_file(path)

Compute the md5 checksum of a file’s contents.

Parameters

path – Path to file

Returns

hexdigest of md5 checksum

scap.utils.mkdir_p(path)

Create directory path.

Parameters

path – The directory path to be created.

scap.utils.open_with_lock(path, mode='r', *args, **kwargs)

Opens the given file and acquires an advisory lock using the open file object. If the mode is read-only (‘r’ or ‘rb’), the lock is acquired as shared, and otherwise acquired as exclusive.

scap.utils.ordered_load(stream, Loader=<class 'yaml.loader.SafeLoader'>, object_pairs_hook=<class 'collections.OrderedDict'>)

Load yaml files and keeping order.

From stackoverflow.com/questions/5121931

:param stream the file object to read :param loader yaml.SafeLoader or its subclasses :object_pairs_hook type of return :return OrderedDict object with the same order of the yaml file

scap.utils.parse_rsync_stats(string: str) → dict

Scans the string looking for text like the following and returns a dictionary with the extracted integer fields.

Note that if no such matching text is found an empty dictionary will be returned.

Number of files: 184,935 (reg: 171,187, dir: 13,596, link: 152) Number of created files: 0 Number of deleted files: 0 Number of regular files transferred: 1 Total file size: 8,756,954,367 bytes Total transferred file size: 815,772 bytes Literal data: 0 bytes Matched data: 815,772 bytes File list size: 4,744,396 File list generation time: 0.517 seconds File list transfer time: 0.000 seconds Total bytes sent: 5,603 Total bytes received: 4,744,454

scap.utils.parse_wmf_version(version: str) → packaging.version.Version

Parses a string like “1.29.0-wmf.4” and returns a packaging.version.Version object representing the version. These objects can be compared using <, <=, >, >=, ==. The special case version of “master” will be treated as a very large version number.

scap.utils.read_first_line_from_file(filename) → str

Reads and returns the first line of the specified file. Whitespace is stripped.

scap.utils.read_wikiversions(directory, realm) → dict

Return a dictionary representing the contents of the realm-specific wikiversions.json file in the specified directory.

Keys are wikidbs, values are “php-<version>”

scap.utils.select_latest_patches(patch_base_dir)

Find the latest /srv/patches/<version> directory. Useful to e.g. populate the patches dir for a new version by carrying over the most recent patches

Returns None if unavailable.

scap.utils.service_exists(service)

Determine if a systemd service unit exists.

scap.utils.sudo_check_call(user, cmd, logger=None, logLevel=10, app=None)

Run a command as a specific user.

Reports stdout/stderr of process to logger during execution.

Returns a string containing stdout/stderr output from the subprocess.

Parameters

• user – User to run command as

• cmd – Command to execute

• logger – Logger to send process output to

• app – Application calling the function, required if the command is scap

Raises

subprocess.CalledProcessError on non-zero process exit

Raises

ValueError if the command is scap and app was not specified

scap.utils.sudo_temp_dir(owner, prefix)

Create a temporary directory and delete it after the block.

Parameters

• owner – Directory owner

• prefix – Temp directory prefix

Returns

Full path to temporary directory

scap.utils.suppress_backtrace()

Context manager that sets the “don’t backtrace” flag on any exception that occurs within context.

Example:

def my_function():

with suppress_backtrace():

some_function_that_may_reasonably_fail()

scap.utils.temp_to_permanent_file(final_filename)

temp_to_permanent_file yields a text stream on a temporary file that is open for writing. If the context body completes without exception, the temp file is renamed to final_filename, atomically replacing any existing file of that name. If an exception is raised during the exception of the body, the temp file is deleted and final_filename remains unaffected.

Example:

with temp_to_permanent_file(“/tmp/important”) as f:

f.write(“Important information”)

scap.utils.var_dump(*args, **kwargs)

dump an object to the console as pretty-printed json

scap.utils.write_file_if_needed(filename, data: str)

Write ‘data’ to ‘filename’ if ‘filename’ doesn’t already have that data in it

Note, the file is written in text mode.

scap.ansi

ANSI escape codes

..seealso:: https://en.wikipedia.org/wiki/ANSI_escape_code

Copyright © 2014-2017 Wikimedia Foundation and Contributors.

This file is part of Scap.

Scap is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, version 3.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program. If not, see <http://www.gnu.org/licenses/>.

scap.ansi.esc(*args)

Get an ANSI escape code.

>>> esc(BG_WHITE, FG_RED, BLINK) == r''
True

Parameters

args – ANSI attributes

Returns

str

scap.ansi.format_ansi(*args)

create an ansi color string from a list of color codes and plain strings.

>>> format_ansi((FG_BLUE,BG_WHITE),'blue on white')                  == 'blue on white'
True

Parameters

args – ANSI color codes and strings of text

Returns

str

scap.ansi.logo(eyes=None, color=True, **colors)

Get the scap logo.

Scappy the scap pig:

        ___ ____
      ⎛   ⎛ ,----
       \  //==--'
  _//| .·//==--'    ____________________________
 _OO≣=-  ︶ ᴹw ⎞_§ ______  ___\ ___\ ,\__ \/ __ \
(∞)_, )  (     |  ______/__  \/ /__ / /_/ / /_/ /
  ¨--¨|| |- (  / _______\____/\___/ \__^_/  .__/
      ««_/  «_/ jgs/bd808               /_/

Ascii art derived from original work by Joan Stark 1 and the speed figlet font 2.

Parameters

• color – Color logo using ANSI escapes

• colors – Alternate colors

Returns

str

1

http://www.oocities.org/spunk1111/farm.htm#pig

2

http://www.jave.de/figlet/fonts/details/speed.html

scap.ansi.reset()

Get the ANSI reset code.

>>> reset() == r''
True

Returns

str

scap.template

Module for working with file templates

Copyright © 2014-2017 Wikimedia Foundation and Contributors.

This file is part of Scap.

Scap is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, version 3.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program. If not, see <http://www.gnu.org/licenses/>.

class scap.template.Template(name, loader, erb_syntax=False, var_file=None, overrides=None, output_format=None)

Adapter class that wraps jinja2 templates.

__init__(name, loader, erb_syntax=False, var_file=None, overrides=None, output_format=None)

_get_file_vars()

Load yaml var file if it exists.

Returns

dict variables for template use

_make_env_args(loader: Dict[str, str], erb_syntax, output_format)

Generate properties to pass to the jinja template.

render() → str

Renders the templates specified by self.name.

It uses the variables sourced from the import yaml file specified by self.var_file

scap.template.get_output_formatter(fmt)

Get output formatter based on desired output format.

scap.template.guess_format(fn)

Guess the output format based on config file extension.

scap.template.yaml_finalize(value)

Output yaml values rather than pythonic values

Logging and Monitoring

scap.log

Helpers for routing and formatting log data.

Copyright © 2014-2017 Wikimedia Foundation and Contributors.

This file is part of Scap.

Scap is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, version 3.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program. If not, see <http://www.gnu.org/licenses/>.

class scap.log.AnsiColorFormatter(fmt=None, datefmt=None, colors=None, colorize='auto')

Colorize output according to logging level.

__init__(fmt=None, datefmt=None, colors=None, colorize='auto')

Parameters

• fmt – Message format string

• datefmt – Time format string

• colors – Dict of {‘levelname’: ANSI SGR parameters}

• colorize – Set to true to colorize messages based on log level. Default ‘auto’ enable color when stderr is a tty or ‘FORCE_COLOR’ is found in the environment.

See also

https://en.wikipedia.org/wiki/ANSI_escape_code

format(record)

Format the specified record as text.

The record’s attribute dictionary is used as the operand to a string formatting operation which yields the returned string. Before formatting the dictionary, a couple of preparatory steps are carried out. The message attribute of the record is computed using LogRecord.getMessage(). If the formatting string uses the time (as determined by a call to usesTime(), formatTime() is called to format the event time. If there is exception information, it is formatted using formatException() and appended to the message.

class scap.log.DeployLogFormatter(fmt=None, datefmt=None, style='%')

Ensure that all deploy.log records contain a host attribute.

format(record)

Format the specified record as text.

The record’s attribute dictionary is used as the operand to a string formatting operation which yields the returned string. Before formatting the dictionary, a couple of preparatory steps are carried out. The message attribute of the record is computed using LogRecord.getMessage(). If the formatting string uses the time (as determined by a call to usesTime(), formatTime() is called to format the event time. If there is exception information, it is formatted using formatException() and appended to the message.

class scap.log.DeployLogHandler(log_file)

Handler for scap/deploy.log.

__init__(log_file)

Open the specified file and use it as the stream for logging.

class scap.log.DiffLogFormatter(fmt=None, datefmt=None, colors=None, colorize='auto')

__init__(fmt=None, datefmt=None, colors=None, colorize='auto')

Parameters

• fmt – Message format string

• datefmt – Time format string

• colors – Dict of {‘levelname’: ANSI SGR parameters}

• colorize – Set to true to colorize messages based on log level. Default ‘auto’ enable color when stderr is a tty or ‘FORCE_COLOR’ is found in the environment.

See also

https://en.wikipedia.org/wiki/ANSI_escape_code

format(record)

Format the specified record as text.

The record’s attribute dictionary is used as the operand to a string formatting operation which yields the returned string. Before formatting the dictionary, a couple of preparatory steps are carried out. The message attribute of the record is computed using LogRecord.getMessage(). If the formatting string uses the time (as determined by a call to usesTime(), formatTime() is called to format the event time. If there is exception information, it is formatted using formatException() and appended to the message.

class scap.log.Filter(criteria, invert=True)

Generic log filter that matches record attributes against criteria.

You can provide either a glob pattern, regular expression, or lambda as each attribute criterion, and invert the logic by passing filter=False.

Examples:

Filter({'name': '*.target.*', 'host': 'scap-target-01'})
Filter({'msg': re.compile('some annoying (message|msg)')})
Filter({'levelno': lambda lvl: lvl < logging.WARNING})
Filter({'name': '*.target.*'}, invert=False)

Equivalent DSL examples:

Filter.loads('name == *.target.* host == scap-target-01')
Filter.loads('msg ~ "some annoying (message|msg)"')
Filter.loads('levelno < WARNING')f
Filter.loads('name == *.target.*', invert=False)

__init__(criteria, invert=True)

append(criteria)

Append the filter with the given criteria.

Parameters

criteria (iter) – Filter criteria

filter(record)

Perform filtering on a given log record.

Parameters

record (LogRecord) – Log record.

isfiltering(attribute)

Whether the filter has criteria for the given attribute.

static loads(expression, invert=True)

Construct a Filter from the given free-form expression.

See Filter for examples.

static parse(expression)

Parse the given filter expression and generates its parts.

Parameters

expression (str) – Filter expression.

Yields

(lhs, op, rhs)

class scap.log.IRCSocketHandler(host, port, timeout=1.0)

Log handler for logmsgbot on #wikimedia-operation.

Sends log events to a tcpircbot server for relay to an IRC channel.

__init__(host, port, timeout=1.0)

Parameters

• host (str) – tcpircbot host

• port (int) – tcpircbot listening port

• timeout (float) – timeout for sending message

emit(record)

Do whatever it takes to actually log the specified logging record.

This version is intended to be implemented by subclasses and so raises a NotImplementedError.

class scap.log.JSONFormatter(fmt=None, datefmt=None, style='%')

Serialize logging output as JSON.

Can be used to maintain logged event structure between the deployment host and remote targets.

format(record)

Format the specified record as text.

The record’s attribute dictionary is used as the operand to a string formatting operation which yields the returned string. Before formatting the dictionary, a couple of preparatory steps are carried out. The message attribute of the record is computed using LogRecord.getMessage(). If the formatting string uses the time (as determined by a call to usesTime(), formatTime() is called to format the event time. If there is exception information, it is formatted using formatException() and appended to the message.

class scap.log.LogstashFormatter(fmt=None, datefmt='%Y-%m-%dT%H:%M:%SZ', log_type='scap')

Format log messages for logstash.

__init__(fmt=None, datefmt='%Y-%m-%dT%H:%M:%SZ', log_type='scap')

Parameters

• fmt – Message format string (not used)

• datefmt – Time format string

• type – Logstash event type

converter()

gmtime([seconds]) -> (tm_year, tm_mon, tm_mday, tm_hour, tm_min,

tm_sec, tm_wday, tm_yday, tm_isdst)

Convert seconds since the Epoch to a time tuple expressing UTC (a.k.a. GMT). When ‘seconds’ is not passed in, convert the current time instead.

If the platform supports the tm_gmtoff and tm_zone, they are available as attributes only.

format(record)

Format a record as a logstash v1 JSON string.

formatException(exc_info)

Format the given exception as a dict.

class scap.log.MuteReporter(name='', expect=0, fd=<_io.TextIOWrapper name='<stderr>' mode='w' encoding='UTF-8'>)

A report that declines to report anything.

__init__(name='', expect=0, fd=<_io.TextIOWrapper name='<stderr>' mode='w' encoding='UTF-8'>)

Parameters

• name – Name of command being monitored

• expect – Number of results to expect

• fd – File handle to write status messages to

• spinner – Cyclical iterator that returns progress spinner.

finish()

Finish tracking progress.

class scap.log.ProgressReporter(name, expect=0, fd=<_io.TextIOWrapper name='<stderr>' mode='w' encoding='UTF-8'>, spinner=None)

Track and display progress of a process.

Report on the status of a multi-step process by displaying the completion percentage and succes, failure and remaining task counts on a single output line.

__init__(name, expect=0, fd=<_io.TextIOWrapper name='<stderr>' mode='w' encoding='UTF-8'>, spinner=None)

Parameters

• name – Name of command being monitored

• expect – Number of results to expect

• fd – File handle to write status messages to

• spinner – Cyclical iterator that returns progress spinner.

add_failure()

Record a failed task completion.

add_success()

Record a sucessful task completion.

expect(count)

Set expected result count.

finish()

Finish tracking progress.

refresh()

Refresh/redraw progress output.

start()

Start tracking progress.

class scap.log.QueueReporter(name, expect=0, fd=<_io.TextIOWrapper name='<stderr>' mode='w' encoding='UTF-8'>, queue=None)

A ProgressReporter which sends its state-changing operations upstream via a queue. It does not generate any output.

__init__(name, expect=0, fd=<_io.TextIOWrapper name='<stderr>' mode='w' encoding='UTF-8'>, queue=None)

Parameters

• name – Name of command being monitored

• expect – Number of results to expect

• fd – File handle to write status messages to

• spinner – Cyclical iterator that returns progress spinner.

add_failure()

Record a failed task completion.

add_success()

Record a sucessful task completion.

expect(count)

Set expected result count.

finish()

Finish tracking progress.

start()

Start tracking progress.

class scap.log.RateLimitedProgressReporter(*args, **kwargs)

The same as ProgressReporter, but doesn’t generate output more than once every max_reporting_interval seconds. The final progress report is always generated.

__init__(*args, **kwargs)

Parameters

• name – Name of command being monitored

• expect – Number of results to expect

• fd – File handle to write status messages to

• spinner – Cyclical iterator that returns progress spinner.

class scap.log.Stats(host, port, logger=None)

A simple StatsD metric client.

It can log measurements and counts to a remote StatsD host. See <https://github.com/etsy/statsd/wiki/Protocol> for details.

__init__(host, port, logger=None)

increment(name, value=1)

Increment a measurement.

timing(name, milliseconds)

Report a timing measurement in milliseconds.

class scap.log.SyslogFormatter(fmt=None, datefmt='%Y-%m-%dT%H:%M:%SZ', log_type='scap')

format(record)

Format a record as a logstash v1 JSON string.

class scap.log.Timer(label, stats=None, logger=None)

Context manager to track and record the time taken to execute a block.

Elapsed time will be recorded to a logger and optionally a StatsD server.

>>> with Timer('example'):
...     time.sleep(0.1)

>>> s = Stats('127.0.0.1', 2003)
>>> with Timer('example', s):
...     time.sleep(0.1)

Sub-interval times can also be recorded using the mark() method.

>>> with Timer('file copy') as t:
...     time.sleep(0.1)
...     x = t.mark('copy phase 1')
...     time.sleep(0.1)
...     y = t.mark('copy phase 2')

__enter__()

Enter the runtime context.

Returns

self

__exit__(exc_type, exc_value, traceback)

Exit the runtime context.

__init__(label, stats=None, logger=None)

Parameters

• label (str) – Label for block (e.g. ‘scap’ or ‘rsync’)

• stats (scap.log.Stats) – StatsD client to record block invocation and duration

_record_elapsed(label, elapsed)

Log the elapsed duration.

Parameters

• label (str) – Label for elapsed time

• elapsed (float) – Elapsed duration

mark(label)

Log the interval elapsed since the last mark call.

Parameters

label (str) – Label for block (e.g. ‘scap’ or ‘rsync’)

class scap.log.Udp2LogHandler(host, port, prefix='scap')

Log handler for udp2log.

__init__(host, port, prefix='scap')

Parameters

• host – Hostname or ip address

• port – Port

• prefix – Line prefix (udp2log destination)

makePickle(record)

Format record as a udp2log packet.

>>> Udp2LogHandler('127.0.0.1', 12345).makePickle(
...     logging.makeLogRecord({'msg':'line1\nline2'}))
'scap line1\nscap line2\n'
>>> Udp2LogHandler('127.0.0.1', 12345).makePickle(
...     logging.makeLogRecord({'msg':'%s12'% ('0'*65500)}))
...     
'scap 00000...00001\n'

scap.log.reporter(message, mute=False)

Instantiate progress reporter

Message

• string that will be displayed to user

scap.log.setup_loggers(cfg, console_level=20, handlers=None)

Setup the logging system.

• Configure the root logger to use DiffLogFormatter

• Optionally add a Udp2LogHandler to send logs to a udp2log server

• Optional add a IRCSocketHandler for the scap.announce log channel to send messages to a tcpircbot server

Parameters

• cfg – Dict of global configuration values

• console_level – Logging level for the local console appender

• handlers – Additional handlers

scap.checks

Deployment checks.

Definitions are typically loaded from YAML of the following format:

checks:

some_unique_check_name:

type: command command: /usr/local/bin/my_special_check stage: promote

some_other_check_name:

type: nrpe command: some_parsed_nrpe_command_name stage: promote

Copyright © 2014-2017 Wikimedia Foundation and Contributors.

This file is part of Scap.

Scap is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, version 3.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program. If not, see <http://www.gnu.org/licenses/>.

class scap.checks.Check(name, stage=None, before=None, after=None, environment=None, group=None, timeout=30.0, command='', shell=False, **opts)

Represent a loaded ‘command’ check.

Parameters

• name – check name

• stage – (deprecated: use “after”, ignored when “after” is set) stage after which to run the check

• before – stage before which to run the check

• after – stage after which to run the check

• environment – environment in which to run checks

• group – deploy group for which to run the check

• timeout – maximum time allowed for check execution, in seconds

• command – check command to run

• shell – If True, a shell is used to execute “command”.

__init__(name, stage=None, before=None, after=None, environment=None, group=None, timeout=30.0, command='', shell=False, **opts)

__repr__()

Return repr(self).

run()

Return a running CheckJob.

validate()

Validate check properties.

exception scap.checks.CheckInvalid

class scap.checks.CheckJob(check)

Represent and control a running check.

A CheckJob begins execution immediately and should be controlled within some kind of poll loop, typically checks.execute.

__init__(check)

Inititalizes a new CheckJob and begins execution.

duration()

Return the current or final job duration.

isfailure()

Return whether this check failed.

kill()

Kill the executing process.

poll()

Read output and polls the process for exit status.

If the process has exited, an (approximate) end time is recorded. This method is non-blocking typically called within an event loop like checks.execute.

timedout()

Whether the job duration has exceeded the job timeout.

wait()

Block for the last stdout/stderr read of the check process.

To ensure the least amount of blocking, this method should only be called within an event loop once poll has signaled that the process has exited.

class scap.checks.OverrideCheck(name, environment)

Represent a loaded ‘override’ check.

__init__(name, environment)

Initialize override check.

scap.checks.checktype(type_of_check)

Class decorator for registering a new check type.

Parameters

type_of_check – type name

scap.checks.execute(checks, logger, concurrency=1)

Execute the given checks in parallel.

Parameters

• checks – iterable of checks.Check objects, or a single checks.Check object.

• logger – logging.Logger to send messages to

• concurrency – level of concurrency

Returns

tuple of the aggregate check success and list of executed checks

Return type

(bool, list)

scap.checks.load(cfg, environment=None)

Load checks from the given config dict.

Parameters

• cfg – config dict

• environment – environment in which to execute checks

scap.checks.register_type(check_type, factory)

Register a new check type and factory.

Parameters

• check_type – type name

• factory – callable type factory

scap.nrpe

NRPE based deployment checks.

Available Icinga/NRPE command definitions must be loaded and registered for use using load or load_directory, and register, before they can be used in checks.yaml configuration.

Example checks.yaml:

checks:

service_endpoints:

type: nrpe command: check_service_endpoints stage: promote

Copyright © 2014-2017 Wikimedia Foundation and Contributors.

This file is part of Scap.

Scap is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, version 3.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program. If not, see <http://www.gnu.org/licenses/>.

class scap.nrpe.NRPECheck(name, stage=None, before=None, after=None, environment=None, group=None, timeout=30.0, command='', shell=False, **opts)

Represent a loaded ‘nrpe’ check.

validate()

Validates that the configured NRPE check is available.

scap.nrpe.load(config)

Load NRPE command definitions from the given configuration.

Parameters

config – NRPE configuration string

Yields

(name, command)

scap.nrpe.load_directory(config_dir)

Load available local NRPE check commands from the given directory.

Parameters

config_dir – directory in which to look for NRPE configuration

Yields

(name, command)

scap.nrpe.register(commands)

Register global NRPE commands for use in check configuration.

Third Party

cdblib

Imported from: python-pure-cdb

Author: David Wilson

License: MIT

Manipulate DJB’s Constant Databases. These are 2 level disk-based hash tables that efficiently handle many keys, while remaining space-efficient.

http://cr.yp.to/cdb.html

When generated databases are only used with Python code, consider using hash() rather than djb_hash() for a tidy speedup.

Note

Minor alterations made to comply with PEP8 style check and to remove attempt to import C implementation of djb_hash. – bd808, 2014-03-04

scap.cdblib.DJB_HASH(s)

Return the value of DJB’s hash function for the given 8-bit string.

>>> py_djb_hash('')
5381
>>> py_djb_hash('
')
177572
>>> py_djb_hash('€')
193278953

scap.cdblib.READ_2_LE4(buffer, /)

Return a tuple containing unpacked values.

Unpack according to the format string Struct.format. The buffer’s size in bytes must be Struct.size.

See help(struct) for more on format strings.

class scap.cdblib.Reader(data, hashfn=<function py_djb_hash>)

A dictionary-like object for reading a Constant Database.

Reader accesses through a string or string-like sequence such as mmap.mmap().

__init__(data, hashfn=<function py_djb_hash>)

Create an instance reading from a sequence and hash keys using hashfn.

>>> Reader(data=b'')
Traceback (most recent call last):
...
OSError: CDB too small
>>> Reader(data=b'a' * 2048) 
<scap.cdblib.Reader object at 0x...>

items()

Like dict.items().

iteritems()

Like dict.iteritems(). Items are returned in insertion order.

scap.cdblib.WRITE_2_LE4()

S.pack(v1, v2, …) -> bytes

Return a bytes object containing values v1, v2, … packed according to the format string S.format. See help(struct) for more on format strings.

class scap.cdblib.Writer(fp, hashfn=<function py_djb_hash>)

Object for building new Constant Databases, and writing them to a seekable file-like object.

__init__(fp, hashfn=<function py_djb_hash>)

Create an instance writing to a file-like object and hash keys.

It uses hashfn to hash keys.

>>> import tempfile
>>> temp_fp = tempfile.TemporaryFile()
>>> Writer(fp=temp_fp, hashfn=py_djb_hash) 
<scap.cdblib.Writer object at 0x...>

finalize()

Write the final hash tables to the output file, and write out its index. The output file remains open upon return.

put(key, value='')

Write a string key/value pair to the output file.

scap.cdblib.py_djb_hash(s)

Return the value of DJB’s hash function for the given 8-bit string.

>>> py_djb_hash('')
5381
>>> py_djb_hash('
')
177572
>>> py_djb_hash('€')
193278953