Python API

Core Application Infrastructure

scap

Wikimedia’s MediaWiki deployment script. Deploys MediaWiki code and configuration to a group of servers via SSH and rsync.

Copyright © 2014-2017 Wikimedia Foundation and Contributors.

This file is part of Scap.

Scap is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, version 3.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program. If not, see <http://www.gnu.org/licenses/>.

class scap.CompileWikiversions(exe_name)[source]

Compile wikiversions.json to wikiversions.php.

main(*extra_args)[source]

Main business logic of the application.

Parsed command line arguments are available in self.arguments. Global configuration is available in self.config. Unparsed command line arguments are passed as positional arguments.

Returns:exit status
class scap.Deploy(exe_name)[source]

Sync new service code across cluster.

Uses local .scaprc as config for each host in cluster

__init__(exe_name)[source]

x.__init__(…) initializes x; see help(type(x)) for signature

_build_deploy_groups()[source]

Build server groups based on configuration server_groups variable

_execute_for_group(stages, group, ignore_failure=False, prompt_user=False)[source]

Executes the given stages across targets of the given group’s subgroups.

Parameters:
  • stages – List of stages to execute
  • group – The targets.DeployGroup for which to execute the stages. Any target host that is unreachable via SSH will be added to the group’s list of excluded hosts.
  • ignore_failure – Whether to keep on rolling past the failure_limit threshold. Note that even with this argument, SSH failure result in the target being excluded from future stage execution.
  • prompt_user – Whether to prompt the user after each subgroup.
_get_stage_name(stage)[source]

Map a stage name to a stage display name.

_load_config()[source]

Set the host directory after the config has been loaded.

_needs_latest_sha1(stages)[source]

Determine whether we expect a new SHA1 to be tagged and deployed

Stages:list of stages being run
_setup_loggers()[source]

Set up additional logging to scap/deploy.log.

checks_setup()[source]

Build info to run checks.

config_deploy_setup(commit)[source]

Generate environment-specific config file and variable template list.

Builds a yaml file that contains: 1. A list of file objects containing template files to be deployed 2. An object containing variables specified in the environment-specific vars.yaml file and inheriting from the vars.yaml file

execute_stage_on_group(stage, group, targets)[source]

Execute a deploy stage for the given group targets.

Parameters:
  • stage – deploy stage.
  • group – deploy group.
  • targets – group targets.
Yields:

(host, status)

main(*extra_args)[source]

Main business logic of the application.

Parsed command line arguments are available in self.arguments. Global configuration is available in self.config. Unparsed command line arguments are passed as positional arguments.

Returns:exit status
class scap.DeployLocal(exe_name)[source]

Command that runs on target hosts.

Responsible for fetching code from the git server, checking out the appropriate revisions, restarting services and running checks.

__init__(exe_name)[source]

x.__init__(…) initializes x; see help(type(x)) for signature

_execute_checks(**kwargs)[source]

Fetch and executes all checks configured for the given stage.

Checks are retrieved from the remote deploy host and cached within tmp.

_get_config_overrides()[source]

Get config information locally or from the deployment host.

If the local [repo]-cache/.config file does not exist, or –refresh-config has been passed explicitly on the command line, then the config is fetched from the deployment server. Otherwise the local config cache is used.

This is useful for things like locally rebuilding config files when a remote var file has changed, and the version deployed from the deployment server has changed, but a particular target has not yet been updated.

_get_remote_overrides()[source]

Grab remote config from git_server.

_load_config()[source]

Load configuration.

_valid_chk(chk, stage, group)[source]

Make sure a check is valid for our current group.

config_deploy()[source]

Render config files.

Grabs the current config yaml file from the deploy git server, and renders the final template inside the repo-cache’s tmp directory.

config_diff()[source]

Render config files from DEPLOY_HEAD and compare each file to the deployed version. This is called by scap deploy –dry-run

fetch()[source]

Fetch the specified revision of the remote repo.

The given repo is cloned into the cache directory and a new working directory for the given revision is created under revs/{rev}.

At the end of this stage, the .in-progress link is created to signal the possibility for future rollback.

finalize(rollback=False, rev=None)[source]

Perform the final deploy actions.

Moves the .done flag to the rev directory, removes the .in-progress flag, and cleans up old revision directories.

main(*extra_args)[source]

Main business logic of the application.

Parsed command line arguments are available in self.arguments. Global configuration is available in self.config. Unparsed command line arguments are passed as positional arguments.

Returns:exit status
promote(rev=None, rev_dir=None, config_deploy=True)[source]

Promote the current deployment.

Switches the current symlink to the current revision directory and restarts the configured service.

Probes the configured service port to measure whether it successfully restarted.

restart_service()[source]

Restart or reload service and check port based on configuration.

rollback()[source]

Performs a rollback to the last deployed revision.

Rollback looks for a .done symlink that points to the revision directory for the last successful deployment. If this link doesn’t exist, a rollback isn’t possible. If it does exist, the current revision directory is replaced with the target of the link and the promote and finalize stages are re-run.

class scap.DeployLog(exe_name)[source]

Tail/filter/output events from the deploy logs.

examples:

deploy-log -v
deploy-log 'host == scap-target-01'
deploy-log 'msg ~ "some important (message|msg)"'
deploy-log 'levelno >= WARNING host == scap-target-*'
_setup_loggers()[source]

Setup logging.

main(*extra_args)[source]

Main business logic of the application.

Parsed command line arguments are available in self.arguments. Global configuration is available in self.config. Unparsed command line arguments are passed as positional arguments.

Returns:exit status
class scap.LockManager(exe_name)[source]

Holds a lock open for a given repository.

examples:

lock 'Testing something, do not deploy'
main(*extra_args)[source]

Main business logic of the application.

Parsed command line arguments are available in self.arguments. Global configuration is available in self.config. Unparsed command line arguments are passed as positional arguments.

Returns:exit status
class scap.MWVersionsInUse(exe_name)[source]

Get a list of the active MediaWiki versions.

main(*extra_args)[source]

Main business logic of the application.

Parsed command line arguments are available in self.arguments. Global configuration is available in self.config. Unparsed command line arguments are passed as positional arguments.

Returns:exit status
class scap.RebuildCdbs(exe_name)[source]

Rebuild localization cache CDB files from the JSON versions.

main(*extra_args)[source]

Main business logic of the application.

Parsed command line arguments are available in self.arguments. Global configuration is available in self.config. Unparsed command line arguments are passed as positional arguments.

Returns:exit status
class scap.RefreshCdbJsonFiles(exe_name)[source]

Create JSON/MD5 files for all CDB files in a directory.

This will put a JSON and MD5 file in /upstream for each CDB file.

This can be combined with rsync and the scap-rebuild-cdbs to push out large CDB files with minimal traffic. CDB files change drastically with small key/value changes, where as JSON files do not, and thus they diff/rdiff much better.

When pushing updates with rsync, this should be run before running rsync. The rsync command should exclude CDB files or at least use -ignore-existing. After the rsync is done, scap-rebuild-cdbs can be run on each server to apply the updates to the CDB files.

main(*extra_args)[source]

Main business logic of the application.

Parsed command line arguments are available in self.arguments. Global configuration is available in self.config. Unparsed command line arguments are passed as positional arguments.

Returns:exit status
class scap.Scap(exe_name)[source]

Deploy MediaWiki to the cluster.

  1. Validate php syntax of wmf-config and multiversion
  2. Sync deploy directory on localhost with staging area
  3. Create/update git repo in staging area
  4. Compile wikiversions.json to php in deploy directory
  5. Update l10n files in staging area
  6. Compute git version information
  7. Commit all changes to local git repo in deploy directory
  8. Ask scap masters to sync with current master
  9. Ask scap proxies to sync with master server
  10. Ask apaches to sync with fastest rsync server
  11. Ask apaches to rebuild l10n CDB files
  12. Update wikiversions.php on localhost
  13. Ask apaches to sync wikiversions.php
  14. Run refreshMessageBlobs.php
  15. Rolling invalidation of all opcache for php 7.x
_before_exit(exit_status)[source]

Do any final cleanup or processing before the application exits.

Called after main() and before sys.exit even when an exception occurs.

Returns:exit status
_handle_exception(ex)[source]

Handle unhandled exceptions and errors.

Returns:exit status
main(*extra_args)[source]

Perform a sync operation to the cluster.

class scap.SecurityPatchCheck(exe_name)[source]

Check if security patches are applied.

class to check if patches in /srv/patches have been applied to the active wikiversions

main(*extra_args)[source]

Main business logic of the application.

Parsed command line arguments are available in self.arguments. Global configuration is available in self.config. Unparsed command line arguments are passed as positional arguments.

Returns:exit status
class scap.SyncCommon(exe_name)[source]

Sync local MediaWiki deployment directory with deploy server state.

main(*extra_args)[source]

Main business logic of the application.

Parsed command line arguments are available in self.arguments. Global configuration is available in self.config. Unparsed command line arguments are passed as positional arguments.

Returns:exit status
class scap.SyncFile(exe_name)[source]

Sync a specific file/directory to the cluster.

_proxy_sync_command()[source]

Synchronization command to run on the proxy hosts.

main(*extra_args)[source]

Perform a sync operation to the cluster.

class scap.SyncL10n(exe_name)[source]

Sync l10n files for a given branch and rebuild cache files.

_proxy_sync_command()[source]

Synchronization command to run on the proxy hosts.

main(*extra_args)[source]

Perform a sync operation to the cluster.

class scap.SyncMaster(exe_name)[source]

Sync local MediaWiki staging directory with deploy server state.

main(*extra_args)[source]

Main business logic of the application.

Parsed command line arguments are available in self.arguments. Global configuration is available in self.config. Unparsed command line arguments are passed as positional arguments.

Returns:exit status
class scap.SyncWikiversions(exe_name)[source]

Rebuild and sync wikiversions.php to the cluster.

_after_sync_common()[source]

Skip this step.

It currently consists only of cache_git_info and this class should attempt to be fast where possible.

_before_cluster_sync()[source]

check for the presence of ExtensionMessages and l10n cache for every branch of mediawiki that is referenced in wikiversions.json to avoid syncing a branch that is lacking these critical files.

class scap.Version(exe_name)[source]
main(*extra_args)[source]

Main business logic of the application.

Parsed command line arguments are available in self.arguments. Global configuration is available in self.config. Unparsed command line arguments are passed as positional arguments.

Returns:exit status

scap.cli

Classes and helpers for creating command line interfaces

Copyright © 2014-2017 Wikimedia Foundation and Contributors.

This file is part of Scap.

Scap is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, version 3.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program. If not, see <http://www.gnu.org/licenses/>.

class scap.cli.Application(exe_name)[source]

Base class for creating command line applications.

__init__(exe_name)[source]

x.__init__(…) initializes x; see help(type(x)) for signature

_assert_auth_sock()[source]

Assert that SSH_AUTH_SOCK is present in the environment.

_assert_current_user(user)[source]

Assert that this program is run as the given user.

_before_exit(exit_status)[source]

Do any final cleanup or processing before the application exits.

Called after main() and before sys.exit even when an exception occurs.

Returns:exit status
_handle_exception(ex)[source]

Handle unhandled exceptions and errors.

Returns:exit status
_load_config()[source]

Load configuration.

_process_arguments(args, extra_args)[source]

Validate and process command line arguments.

Default behavior is to abort the application with an error if any unparsed arguments were found.

Returns:Tuple of (args, extra_args) after processing
_run_as(user)[source]

Ensure that this program is run as the given user.

_setup_environ()[source]

Setup shell environment.

_setup_loggers()[source]

Setup logging.

active_wikiversions(source_tree='deploy')[source]

Get an ordered collection of active MediaWiki versions.

Parameters:source_tree – Source tree to read file from: ‘deploy’ or ‘stage’
Returns:collections.OrderedDict of {version:wikidb} values sorted by version number in ascending order
announce(*args)[source]

Announce a message to broadcast listeners.

Emits a logging event to the ‘scap.announce’ logger which can be configured to broadcast messages via irc or another real-time notification channel.

Announcements can be disabled by using ‘–no-log-message’ in the command invocation (e.g. scap sync-file –no-log-message foo.php). In this case the log event will still be emitted to the normal logger as though self.get_logger().info() was used instead of self.announce().

get_duration()[source]

Get the elapsed duration in seconds.

get_keyholder_key()[source]

Get the public key for IdentityFile use in ssh.

get_lock_file()[source]

Get the path to scap.lock

get_logger()[source]

Lazy getter for a logger instance.

get_master_list()[source]

Get list of deploy master hostnames that should be updated before the rest of the cluster.

get_script_path()[source]

Qualify the path to the scap script.

handle_keyboard_interrupt()[source]

Handle ctrl-c from interactive user.

Returns:exit status
main(*extra_args)[source]

Main business logic of the application.

Parsed command line arguments are available in self.arguments. Global configuration is available in self.config. Unparsed command line arguments are passed as positional arguments.

Returns:exit status
classmethod run()[source]

Construct and run an application.

Calls sys.exit with exit status returned by application by default. Setting exit to False will instead return the class instance and it’s exit status. This would generally only be done when testing.

Parameters:
  • cls – Class to create and run
  • argv – Command line arguments
Returns:

Tuple of class instance and exit status when not exiting

scap.cli.all_commands()[source]

return a list of all commands that have been registered with the command() decorator.

scap.cli.argument(*args, **kwargs)[source]

Decorator used to declare a command line argument on an Application

@scap.cli.argument(option_flags..[,action='store'][,nargs=1] [,const=None][,default][,type=str][,choices][,required=False][,help] [,dest])[source]

Maps a command line argument to the decorated class or method.

Parameters:
  • option_flags (str) – One or more option flags associated with this argument, e.g. ‘-a’, ‘–arg’
  • action – the action associated with this argument. e.g. ‘store_true’
  • default – The default value for this argument if not specified by the user.
  • help (str) – Short description of this argument, displayed in --help text.
  • choices (list) – List possible values for this argument. If specified, argument values will be validated for you and --help will list the possible choices for the user.
  • required (bool) – True if your argument is required.
  • type (type) – The type of value accepted by your argument, e.g. int or file.
  • nargs (int, str) – The number of values accepted by this argument.
scap.cli.command(*args, **kwargs)[source]
@scap.cli.command(command_name, help="help text"[, subcommands=False])[source]

Map a scap sub-command to the decorated class.

Parameters:
  • command_name (str) – The name of the sub-command
  • help (str or None) – A summary of your command to be displayed in –help text
Raises:

ValueError – if there is already a command named command_name

Usage Example:

import scap.cli

@cli.command('hello', help='prints "hello world" and exits')
class HelloCommand(cli.Application):
    @cli.argument('--goodbye', action='store_true',
                  help='Say goodbye instead.')
    def main(extra_args):
        if self.arguments.goodbye:
            print('Goodbye, cruel world.')
        else:
            print('Hello, world.')
scap.cli.subcommand(name=None)[source]
@scap.cli.subcommand(command_name)[source]

Define an argparse subcommand by decorating a method on your cli.Application subclass.

Use this decorator when you want to have a subcommand on a method other than ‘main’

In order for this to have any affect, your cli.Application must be decorated with subcommands=True (see example below).

Usage Example:

import scap.cli

@cli.command('hello', subcommands=True,
             help='prints "hello world" and exits',)
class HelloCommand(cli.Application):
    @cli.subcommand('world')
    def world_subcommand(extra_args):
        print('hello world')

scap.arg

Helpers for creating a fancy argparser. Most of the externally useful API for command line arg parsing is found in scap.cli

Author:Tyler Cipriani <thcipriani@wikimedia.org>
Author:Mukunda Modell <mmodell@wikimedia.org>
Copyright:Wikimedia Foundation, Inc.
License:GPL v3.0

Parts of the shell argument completion code in this file is derived from python-selfcompletion.

Author:David Barnett <davidbarnett2@gmail.com>
License:BSD

Copyright © 2014-2017 Wikimedia Foundation and Contributors.

This file is part of Scap.

Scap is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, version 3.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program. If not, see <http://www.gnu.org/licenses/>.

class scap.arg.ScapArgParser(*args, **kwargs)[source]

Scap argparse subclass

Created to allow for easier, scripted, autocompletion

__init__(*args, **kwargs)[source]

x.__init__(…) initializes x; see help(type(x)) for signature

get_valid_next_words(words, more_valid_words=None)[source]

get completion words for cli auto-complete

class scap.arg.ScapHelpFormatter(prog, indent_increment=2, max_help_position=24, width=None)[source]

Formatter that respects argparse.SUPPRESS for subparser actions.

class scap.arg._ScapAutoCompleteAction[source]
__call__(...) <==> x(...)[source]
__init__()[source]

x.__init__(…) initializes x; see help(type(x)) for signature

scap.arg.build_parser()[source]

Build an argument parser for all cli.Application’s.

scap.arg.build_subparser(cmd, parser, global_parser)[source]

Append subparsers to cli.Application’s argparser using decorators.

scap.arg.extract_help_from_object(obj)[source]

Extract help information from the object’s docblock

scap.arg.get_global_parser()[source]

Add standard arguments to argparser.

These arguments should be present on all subparsers.

..info::

The other option with these commands would be to make them into top-level flags for scap; however, that ends up feeling clunky with commands like:

scap –verbose sync

Or

scap -e beta deploy –repo mockbase/deploy
scap.arg.is_dir(string)[source]

represents a cli argument which accepts only a valid directory name

scap.arg.is_version(string)[source]

represents a cli argument which accepts a mediawiki branch version

scap.config

Configuration management

Copyright © 2014-2017 Wikimedia Foundation and Contributors.

This file is part of Scap.

Scap is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, version 3.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program. If not, see <http://www.gnu.org/licenses/>.

scap.config.coerce_value(key, value)[source]

Coerce the given value based on the default config type.

scap.config.load(cfg_file=None, environment=None, overrides=None)[source]

Load configuration.

A configuration file consists of sections, led by a [section] header and followed by name: value entries. Lines beginning with '#' are ignored and may be used to provide comments.

A configuration file can contain multiple sections. The configuration object is populated with values from the global section and additional sections based on the fully qualified domain name of the local host. For example, on the host tin.eqiad.wmnet the final value for a given setting would be the first value found in sections: tin.eqiad.wmnet, eqiad.wmnet, wmnet or global. Sections not present in the configuration file will be ignored.

Configuration values are loaded from a file specified by the -c or --conf command-line options or from the default locations with the following hierarchy, sorted by override priority:

  1. $(pwd)/scap/environments/<environment>/scap.cfg or $(pwd)/scap/scap.cfg (if no environment was specified)
  2. /etc/scap.cfg

For example, if a configuration parameter is set in $(pwd)/scap/scap.cfg and that same parameter is set in /etc/scap.cfg the value for that parameter set in $(pwd)/scap/scap.cfg will be used during execution.

Parameters:
  • cfg_file – Alternate configuration file
  • environment – the string path under which scap.cfg is found
  • overrides – Dict of configuration values
Returns:

dict of configuration values

scap.config.multi_value(str_value)[source]

Given a string that’s got commas, turn it into a list

Parameters:str_value – Random thing the user typed in config
scap.config.override_config(config, overrides=None)[source]

Override values in a config with type-coerced values.

scap.terminal

Text terminal output utilities

Copyright © 2014-2017 Wikimedia Foundation and Contributors.

This file is part of Scap.

Scap is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, version 3.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program. If not, see <http://www.gnu.org/licenses/>.

class scap.terminal.TerminalIO(out=None, autoflush=True)[source]

TerminalIO represents a terminal (pty) and provides several convenience methods for outputting terminal control sequences. Much of this code was derived from the blessed library <https://github.com/jquast/blessed>. The API for TerminalIO is different from the blessed Terminal API.

License:

Copyright (c) 2016 Mukunda Modell <mmodell@wikimedia.org> Copyright (c) 2014 Jeff Quast Copyright (c) 2011 Erik Rose

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

__init__(out=None, autoflush=True)[source]

x.__init__(…) initializes x; see help(type(x)) for signature

_height_and_width()[source]

Return a tuple of (terminal height, terminal width). If stream or sys.__stdout__ is not a tty or does not support fcntl.ioctl() of termios.TIOCGWINSZ, a window size of 80 columns by 25 rows is returned for any values not represented by environment variables LINES and COLUMNS, which is the default text mode of IBM PC compatibles. :rtype: WINSZ WINSZ is a collections.namedtuple instance, whose structure directly maps to the return value of the termios.TIOCGWINSZ ioctl return value. The return parameters are: * ws_row: width of terminal by its number of character cells. * ws_col: height of terminal by its number of character cells. * ws_xpixel: width of terminal by pixels (not accurate). * ws_ypixel: height of terminal by pixels (not accurate).

static _winsize(fd)[source]

Return named tuple describing size of the terminal by fd. If the given platform does not have modules termios, fcntl, or tty, window size of 80 columns by 25 rows is always returned. :arg int fd: file descriptor queries for its window size. :raises IOError: the file descriptor fd is not a terminal. :rtype: WINSZ WINSZ is a collections.namedtuple instance, whose structure directly maps to the return value of the termios.TIOCGWINSZ ioctl return value. The return parameters are: * ws_row: width of terminal by its number of character cells. * ws_col: height of terminal by its number of character cells. * ws_xpixel: width of terminal by pixels (not accurate). * ws_ypixel: height of terminal by pixels (not accurate).

cleanup()[source]

call cleanup callbacks registered by other modules

close()[source]

Flush and close the IO object.

This method has no effect if the file is already closed.

does_styling

Read-only property: Whether this class instance may emit sequences. :rtype: bool

flush()[source]

Flush write buffers, if applicable.

This is not implemented for read-only and non-blocking streams.

height

Read-only property: Height of the terminal (in number of lines). :rtype: int

is_a_tty

Read-only property: Whether stream is a terminal. :rtype: bool

kind

Read-only property: Terminal kind determined on class initialization. :rtype: str

width

Read-only property: Width of the terminal (in number of columns). :rtype: int

write(*args)[source]

Write string to stream. Returns the number of characters written (which is always equal to the length of the string).

class scap.terminal.WINSZ[source]

Structure represents return value of termios.TIOCGWINSZ. .. py:attribute:: ws_row rows, in characters .. py:attribute:: ws_col columns, in characters .. py:attribute:: ws_xpixel horizontal size, pixels .. py:attribute:: ws_ypixel vertical size, pixels

_BUF = '\x00\x00\x00\x00\x00\x00\x00\x00'

buffer of termios structure appropriate for ioctl argument

_FMT = 'hhhh'

format of termios structure

Built-in scap command classes

scap.main

Command wrappers for scap tasks

Copyright © 2014-2017 Wikimedia Foundation and Contributors.

This file is part of Scap.

Scap is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, version 3.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program. If not, see <http://www.gnu.org/licenses/>.

class scap.main.AbstractSync(exe_name)[source]

Base class for applications that want to sync one or more files from the deployment server to the rest of the cluster.

__init__(exe_name)[source]

x.__init__(…) initializes x; see help(type(x)) for signature

_apache_sync_command(proxies)[source]

Synchronization command to run on the apache hosts.

Parameters:proxies – List of proxy hostnames
_get_api_canary_list()[source]

Get list of MediaWiki api canaries.

_get_app_canary_list()[source]

Get list of MediaWiki api canaries.

_get_canary_list()[source]

Get list of MediaWiki canary hostnames.

_get_proxy_list()[source]

Get list of sync proxy hostnames that should be updated before the rest of the cluster.

_get_target_list()[source]

Get list of hostnames that should be updated from the proxies.

_git_repo()[source]

Flatten deploy directory into shared git repo.

_invalidate_opcache(target_hosts=None, filename=None)[source]

Invalidate opcache

_master_sync_command()[source]

Synchronization command to run on the master hosts.

_proxy_sync_command()[source]

Synchronization command to run on the proxy hosts.

_restart_php()[source]

Check if php-fpm opcache is full, if so restart php-fpm

_sync_common()[source]

Sync stage_dir to deploy_dir on the deployment host.

_sync_masters()[source]

Sync the staging directory across all deploy master servers.

canary_checks(canaries=None, timer=None)[source]

Run canary checks

Parameters:
  • canaries – Iterable of canary servers to check
  • timer – log.Timer
Raises:

RuntimeError – on canary check failure

get_keyholder_key()[source]

Returns scap2-specific deploy key

This way we can set a key in the default scap config without having all non-scap2 repos inherit that configuration.

increment_stat(stat, all_stat=True, value=1)[source]

Increment a stat in deploy.*

Parameters:
  • stat – String name of stat to increment
  • all_stat – Whether to increment deploy.all as well
  • value – How many to increment by, default of 1 is normal
main(*extra_args)[source]

Perform a sync operation to the cluster.

master_only_cmd(timer, cmd)[source]

Run a command on all other master servers than the one we’re on

Parameters:
  • timer – String name to use in timer/logging
  • cmd – List of command/parameters to be executed
sync_canary(canaries=None)[source]

Sync canary hosts

Parameters:canaries – Iterable of canary servers to sync
class scap.main.CompileWikiversions(exe_name)[source]

Compile wikiversions.json to wikiversions.php.

main(*extra_args)[source]

Main business logic of the application.

Parsed command line arguments are available in self.arguments. Global configuration is available in self.config. Unparsed command line arguments are passed as positional arguments.

Returns:exit status
class scap.main.LockManager(exe_name)[source]

Holds a lock open for a given repository.

examples:

lock 'Testing something, do not deploy'
main(*extra_args)[source]

Main business logic of the application.

Parsed command line arguments are available in self.arguments. Global configuration is available in self.config. Unparsed command line arguments are passed as positional arguments.

Returns:exit status
class scap.main.MWVersionsInUse(exe_name)[source]

Get a list of the active MediaWiki versions.

main(*extra_args)[source]

Main business logic of the application.

Parsed command line arguments are available in self.arguments. Global configuration is available in self.config. Unparsed command line arguments are passed as positional arguments.

Returns:exit status
class scap.main.RebuildCdbs(exe_name)[source]

Rebuild localization cache CDB files from the JSON versions.

main(*extra_args)[source]

Main business logic of the application.

Parsed command line arguments are available in self.arguments. Global configuration is available in self.config. Unparsed command line arguments are passed as positional arguments.

Returns:exit status
class scap.main.RefreshCdbJsonFiles(exe_name)[source]

Create JSON/MD5 files for all CDB files in a directory.

This will put a JSON and MD5 file in /upstream for each CDB file.

This can be combined with rsync and the scap-rebuild-cdbs to push out large CDB files with minimal traffic. CDB files change drastically with small key/value changes, where as JSON files do not, and thus they diff/rdiff much better.

When pushing updates with rsync, this should be run before running rsync. The rsync command should exclude CDB files or at least use -ignore-existing. After the rsync is done, scap-rebuild-cdbs can be run on each server to apply the updates to the CDB files.

main(*extra_args)[source]

Main business logic of the application.

Parsed command line arguments are available in self.arguments. Global configuration is available in self.config. Unparsed command line arguments are passed as positional arguments.

Returns:exit status
class scap.main.Scap(exe_name)[source]

Deploy MediaWiki to the cluster.

  1. Validate php syntax of wmf-config and multiversion
  2. Sync deploy directory on localhost with staging area
  3. Create/update git repo in staging area
  4. Compile wikiversions.json to php in deploy directory
  5. Update l10n files in staging area
  6. Compute git version information
  7. Commit all changes to local git repo in deploy directory
  8. Ask scap masters to sync with current master
  9. Ask scap proxies to sync with master server
  10. Ask apaches to sync with fastest rsync server
  11. Ask apaches to rebuild l10n CDB files
  12. Update wikiversions.php on localhost
  13. Ask apaches to sync wikiversions.php
  14. Run refreshMessageBlobs.php
  15. Rolling invalidation of all opcache for php 7.x
_before_exit(exit_status)[source]

Do any final cleanup or processing before the application exits.

Called after main() and before sys.exit even when an exception occurs.

Returns:exit status
_handle_exception(ex)[source]

Handle unhandled exceptions and errors.

Returns:exit status
main(*extra_args)[source]

Perform a sync operation to the cluster.

class scap.main.SecurityPatchCheck(exe_name)[source]

Check if security patches are applied.

class to check if patches in /srv/patches have been applied to the active wikiversions

main(*extra_args)[source]

Main business logic of the application.

Parsed command line arguments are available in self.arguments. Global configuration is available in self.config. Unparsed command line arguments are passed as positional arguments.

Returns:exit status
class scap.main.SyncCommon(exe_name)[source]

Sync local MediaWiki deployment directory with deploy server state.

main(*extra_args)[source]

Main business logic of the application.

Parsed command line arguments are available in self.arguments. Global configuration is available in self.config. Unparsed command line arguments are passed as positional arguments.

Returns:exit status
class scap.main.SyncFile(exe_name)[source]

Sync a specific file/directory to the cluster.

_proxy_sync_command()[source]

Synchronization command to run on the proxy hosts.

main(*extra_args)[source]

Perform a sync operation to the cluster.

class scap.main.SyncL10n(exe_name)[source]

Sync l10n files for a given branch and rebuild cache files.

_proxy_sync_command()[source]

Synchronization command to run on the proxy hosts.

main(*extra_args)[source]

Perform a sync operation to the cluster.

class scap.main.SyncMaster(exe_name)[source]

Sync local MediaWiki staging directory with deploy server state.

main(*extra_args)[source]

Main business logic of the application.

Parsed command line arguments are available in self.arguments. Global configuration is available in self.config. Unparsed command line arguments are passed as positional arguments.

Returns:exit status
class scap.main.SyncWikiversions(exe_name)[source]

Rebuild and sync wikiversions.php to the cluster.

_after_sync_common()[source]

Skip this step.

It currently consists only of cache_git_info and this class should attempt to be fast where possible.

_before_cluster_sync()[source]

check for the presence of ExtensionMessages and l10n cache for every branch of mediawiki that is referenced in wikiversions.json to avoid syncing a branch that is lacking these critical files.

class scap.main.Version(exe_name)[source]
main(*extra_args)[source]

Main business logic of the application.

Parsed command line arguments are available in self.arguments. Global configuration is available in self.config. Unparsed command line arguments are passed as positional arguments.

Returns:exit status

Plug-in scap sub-command classes

scap.plugins

Scap plugin architecture

scap.plugins.find_plugins(plugin_dirs)[source]

Get a list of all plugins found in in plugin_dirs

Parameters:plugin_dirs (list) – directories to search for plugins
Returns:list of all plugin commands found in plugin_dirs
scap.plugins.load_plugins([plugin_dir])[source]

load scap plugin modules.

Parameters:plugin_dir (str) – an additional location to search

Copyright © 2014-2017 Wikimedia Foundation and Contributors.

This file is part of Scap.

Scap is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, version 3.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program. If not, see <http://www.gnu.org/licenses/>.

class scap.plugins.Say(exe_name)[source]

Scap propaganda of the lowest order.

main(*extra_args)[source]

Main business logic of the application.

Parsed command line arguments are available in self.arguments. Global configuration is available in self.config. Unparsed command line arguments are passed as positional arguments.

Returns:exit status

Command Execution

scap.cmd

command execution

Copyright © 2014-2017 Wikimedia Foundation and Contributors.

This file is part of Scap.

Scap is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, version 3.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program. If not, see <http://www.gnu.org/licenses/>.

class scap.cmd.Command(*cmds)[source]

Command is used to store a parameterized shell command for reuse by cluster_ssh and other utilities. The idea is that we store the command, along with any arguments that need to be filled in, then we can call the command later just like a normal python function, which returns the shell command to be ran.

Usage Example:

import scap.cmd as cmd

ssh = cmd.Command('/usr/bin/ssh', cmd.arg('user', '-oUser={}'))
sudo = cmd.Command('sudo', cmd.arg('user', '-u {}'), '-n', '--')
ssh('some.host', sudo('remote_cmd', 'some', 'args', user='sudo_user'),
    user='ssh_user')

# result:

['/usr/bin/ssh',
 '-oUser=ssh_user',
 'some.host',
 'sudo',
 '-u sudo_user',
 '-n',
 '--',
 'remote_cmd',
 'some',
 'args']
__call__(...) <==> x(...)[source]
__init__(*cmds)[source]

x.__init__(…) initializes x; see help(type(x)) for signature

class scap.cmd.arg(name, cmd)[source]

Represent a named parameter that will be passed to an instance of Command.

For example, arg('user', '-u {}') creates an arg called ‘user’ which will evaluate to "-u luser" when the command is evaluated with some_command(user='luser')

Parameters:
  • name – The name of the parameter, used to specify it’s value later when the command is evaluated.
  • cmd – string, the argument text format string.
__call__(...) <==> x(...)[source]
__init__(name, cmd)[source]

x.__init__(…) initializes x; see help(type(x)) for signature

required(required=True)[source]

Mark this argument as required

scap.ssh

This module provides functions for running commands on remote hosts via SSH.

Copyright © 2014-2017 Wikimedia Foundation and Contributors.

This file is part of Scap.

Scap is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, version 3.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program. If not, see <http://www.gnu.org/licenses/>.

class scap.ssh.JSONOutputHandler(host)[source]

Deserialize and log structured JSON output from hosts.

Any non-structured output is stored for future handling.

__init__(host)[source]

x.__init__(…) initializes x; see help(type(x)) for signature

accept(output)[source]

Extract and deserializes line-wise JSON from the given output.

Any non-JSON is stored in self.output.

lines(output)[source]

Generate each line of the given output.

Reconstructs partial lines using the leftovers from previous calls.

class scap.ssh.Job(**kwargs)[source]

Execute a job on a group of remote hosts via ssh.

__init__(**kwargs)[source]

x.__init__(…) initializes x; see help(type(x)) for signature

command(command)[source]

Set command to run.

get_logger()[source]

Lazy getter for a logger instance.

hosts(hosts)[source]

Set hosts to run command on.

progress(reporter)[source]

Set the reporter used when reporting progress.

run(batch_size=80)[source]

Run the job, report progress, and return success/failed counts.

Returns:(ok, failed) counts of successful/failed hosts
Raises:RuntimeError if command has not been set
run_with_status(batch_size=80)[source]

Run the job, report progress, and yield host/status as execution completes.

Yields:(host, status)
Raises:RuntimeError if command has not been set
shuffle()[source]

Randomize order of target hosts.

class scap.ssh.OutputHandler(host)[source]

Standard handler for SSH command output from hosts.

Simply stores output as a string for future handling.

__init__(host)[source]

x.__init__(…) initializes x; see help(type(x)) for signature

scap.ssh.cluster_ssh(hosts, command, user=None, key=None, limit=80, max_fail=None, output_handler=None, verbose=False)[source]

Run a command via SSH on multiple hosts concurrently.

scap.tasks

Contains functions implementing scap tasks

Copyright © 2014-2017 Wikimedia Foundation and Contributors.

This file is part of Scap.

Scap is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, version 3.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program. If not, see <http://www.gnu.org/licenses/>.

scap.tasks._call_rebuildLocalisationCache(wikidb, out_dir, use_cores=1, lang=None, force=False, quiet=False)[source]

Helper for update_localization_cache.

Parameters:
  • wikidb – Wiki running given version
  • out_dir – The output directory
  • use_cores – The number of cores to run in
  • lang – The –lang option, or None to omit
  • force – Whether to pass –force
  • quiet – Whether to pass –quiet
scap.tasks.cache_git_info(version, cfg)[source]

Create JSON cache files of git branch information.

Parameters:
  • version – MediaWiki version (eg ‘1.23wmf15’)
  • cfg – Dict of global configuration values
Raises:

IOError if version directory is not found

scap.tasks.check_patch_files(version, cfg)[source]

Check to see if there are unmerged patch files from /srv/patches for a given revision.

Parameters:
  • version – MediaWiki version string (e.g., ‘1.27.0-wmf.8’)
  • cfg – Scap configuration dict
scap.tasks.clear_message_blobs(*args, **kwargs)[source]

Clear MessageBlobStore cache on all wikis

Parameters:logger – logger instance
scap.tasks.compile_wikiversions(*args, **kwargs)[source]

Validate and compile the wikiversions.json file.

  1. Find the realm specific filename for wikiversions.json in staging area
  2. Validate that all versions mentioned in the json exist as directories in the staging area
  3. Validate that all wikis listed in the realm specific all.dblist exist in the json
  4. Create a temporary CDB file from the json contents
  5. Create a temporary php file from the json contents
  6. Atomically rename the temporary php to the realm specific wikiversions.php filename
Parameters:
  • source_tree – Source tree to read file from: ‘deploy’ or ‘stage’
  • cfg – Dict of global configuration values
scap.tasks.endpoint_canary_checks(canaries, url, spec_path='/spec.yaml', cores=2)[source]

Run service-checker-swagger canary checks on test application servers.

Parameters:
  • canaries – list, canaries to check
  • url – url to pass to service-checker-swagger
  • spec_path – url to pass to service-checker-swagger
  • cores – number of processor cores to use
scap.tasks.get_old_wikiversions(versions, keep=2, keep_static=5)[source]

Get lists of old MediaWiki versions to be removed

Parameters:
  • keep=2 – Number of branches for which we want to keep everything
  • keep_static=5 – Number of weeks to keep static assets
Returns:

tuple of lists of old wikiversions

scap.tasks.get_wikiversions_ondisk(directory)[source]

Get checked-out wikiversions in a directory.

Finds wikiversions in a directory and does its best to determine the date of the oldest reflog for that branch (non recursive)

Returns:list of tuples like:: [(/path/to/php-1.29.0-wmf.17, <DateCreated>)]`
scap.tasks.handle_services(services, require_valid_service=False)[source]

Take a comma-separated list of services, and restart each of them.

The idea is to take a string directly from the scap.cfg file that looks like:

jobrunner, jobchron = reload

and be able to determine what to do with that list.

scap.tasks.logstash_canary_checks(canaries, service, threshold, logstash, delay, cores=2)[source]

Run the logstash canary checks on test application servers.

Parameters:
  • canaries – list, canaries to check
  • threshold – float, average log multiple at which to fail
  • service – string, name of the service to check
  • logstash – string, logstash server
  • verbose – bool, verbose output
  • delay – float, time between deploy and now
  • cores – number of processor cores to use
scap.tasks.merge_cdb_updates(*args, **kwargs)[source]

Update l10n CDB files using JSON data.

Parameters:
  • directory – L10n cache directory
  • pool_size – Number of parallel processes to use
  • trust_mtime – Trust file modification time?
  • mute – Disable progress indicator
scap.tasks.refresh_cdb_json_file(file_path)[source]

Rebuild json file from cdb file.

  1. Check md5 file saved in upstream against md5 of cdb file
  2. Read cdb file to dict
  3. Write dict to named temporary file
  4. Change permissions on named temporary file
  5. Overwrite upstream json file
  6. Write upstream md5 file
scap.tasks.refresh_cdb_json_files(in_dir, pool_size, verbose)[source]

Update json files from corresponding cdb file in parallel.

Parameters:
  • in_dir – directory containing cdb files
  • pool_size – number of “threads” to use
  • verbose – output verbosely
scap.tasks.sync_common(*args, **kwargs)[source]

Sync local deploy dir with upstream rsync server’s copy.

Rsync from server::common to the local deploy directory. If a list of servers is given in sync_from we will attempt to select the “best” one to sync from. If no servers are given or all servers given have issues we will fall back to using the server named by master_rsync in the configuration data.

Parameters:
  • cfg – Dict of global configuration values.
  • include – List of rsync include patterns to limit the sync to. If None is given the entire common module on the target rsync server will be transferred. Rsync syntax for syncing a directory is <dirname>/***.
  • sync_from – List of rsync servers to fetch from.
scap.tasks.sync_master(*args, **kwargs)[source]

Sync local staging dir with upstream rsync server’s copy.

Rsync from server::common to the local staging directory.

Parameters:
  • cfg – Dict of global configuration values.
  • master – Master server to sync with
  • verbose – Enable verbose logging?
scap.tasks.sync_wikiversions(hosts, cfg, key=None)[source]

Rebuild and sync wikiversions.php to the cluster.

Parameters:
  • hosts – List of hosts to sync to
  • cfg – Dict of global configuration values
scap.tasks.update_l10n_cdb(*args, **kwargs)[source]

Update a localization CDB database.

Parameters:
  • cache_dir – L10n cache directory
  • cdb_file – L10n CDB database
  • trust_mtime – Trust file modification time?
scap.tasks.update_l10n_cdb_wrapper(*args, **kwargs)[source]

Wrapper for update_l10n_cdb to be used in contexts where only a single argument can be provided.

Parameters:args – Sequence of arguments to pass to update_l10n_cdb
scap.tasks.update_localization_cache(*args, **kwargs)[source]

Update the localization cache for a given MW version.

Parameters:
  • version – MediaWiki version
  • wikidb – Wiki running given version
  • verbose – Provide verbose output
  • cfg – Global configuration

Git Repository Deployment

scap.deploy

Command wrappers for deploy tasks

Copyright © 2014-2017 Wikimedia Foundation and Contributors.

This file is part of Scap.

Scap is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, version 3.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program. If not, see <http://www.gnu.org/licenses/>.

class scap.deploy.Deploy(exe_name)[source]

Sync new service code across cluster.

Uses local .scaprc as config for each host in cluster

__init__(exe_name)[source]

x.__init__(…) initializes x; see help(type(x)) for signature

_build_deploy_groups()[source]

Build server groups based on configuration server_groups variable

_execute_for_group(stages, group, ignore_failure=False, prompt_user=False)[source]

Executes the given stages across targets of the given group’s subgroups.

Parameters:
  • stages – List of stages to execute
  • group – The targets.DeployGroup for which to execute the stages. Any target host that is unreachable via SSH will be added to the group’s list of excluded hosts.
  • ignore_failure – Whether to keep on rolling past the failure_limit threshold. Note that even with this argument, SSH failure result in the target being excluded from future stage execution.
  • prompt_user – Whether to prompt the user after each subgroup.
_get_stage_name(stage)[source]

Map a stage name to a stage display name.

_load_config()[source]

Set the host directory after the config has been loaded.

_needs_latest_sha1(stages)[source]

Determine whether we expect a new SHA1 to be tagged and deployed

Stages:list of stages being run
_setup_loggers()[source]

Set up additional logging to scap/deploy.log.

checks_setup()[source]

Build info to run checks.

config_deploy_setup(commit)[source]

Generate environment-specific config file and variable template list.

Builds a yaml file that contains: 1. A list of file objects containing template files to be deployed 2. An object containing variables specified in the environment-specific vars.yaml file and inheriting from the vars.yaml file

execute_stage_on_group(stage, group, targets)[source]

Execute a deploy stage for the given group targets.

Parameters:
  • stage – deploy stage.
  • group – deploy group.
  • targets – group targets.
Yields:

(host, status)

main(*extra_args)[source]

Main business logic of the application.

Parsed command line arguments are available in self.arguments. Global configuration is available in self.config. Unparsed command line arguments are passed as positional arguments.

Returns:exit status
exception scap.deploy.DeployGroupFailure[source]

Signal that a particular deploy group failed

class scap.deploy.DeployLocal(exe_name)[source]

Command that runs on target hosts.

Responsible for fetching code from the git server, checking out the appropriate revisions, restarting services and running checks.

__init__(exe_name)[source]

x.__init__(…) initializes x; see help(type(x)) for signature

_execute_checks(**kwargs)[source]

Fetch and executes all checks configured for the given stage.

Checks are retrieved from the remote deploy host and cached within tmp.

_get_config_overrides()[source]

Get config information locally or from the deployment host.

If the local [repo]-cache/.config file does not exist, or –refresh-config has been passed explicitly on the command line, then the config is fetched from the deployment server. Otherwise the local config cache is used.

This is useful for things like locally rebuilding config files when a remote var file has changed, and the version deployed from the deployment server has changed, but a particular target has not yet been updated.

_get_remote_overrides()[source]

Grab remote config from git_server.

_load_config()[source]

Load configuration.

_valid_chk(chk, stage, group)[source]

Make sure a check is valid for our current group.

config_deploy()[source]

Render config files.

Grabs the current config yaml file from the deploy git server, and renders the final template inside the repo-cache’s tmp directory.

config_diff()[source]

Render config files from DEPLOY_HEAD and compare each file to the deployed version. This is called by scap deploy –dry-run

fetch()[source]

Fetch the specified revision of the remote repo.

The given repo is cloned into the cache directory and a new working directory for the given revision is created under revs/{rev}.

At the end of this stage, the .in-progress link is created to signal the possibility for future rollback.

finalize(rollback=False, rev=None)[source]

Perform the final deploy actions.

Moves the .done flag to the rev directory, removes the .in-progress flag, and cleans up old revision directories.

main(*extra_args)[source]

Main business logic of the application.

Parsed command line arguments are available in self.arguments. Global configuration is available in self.config. Unparsed command line arguments are passed as positional arguments.

Returns:exit status
promote(rev=None, rev_dir=None, config_deploy=True)[source]

Promote the current deployment.

Switches the current symlink to the current revision directory and restarts the configured service.

Probes the configured service port to measure whether it successfully restarted.

restart_service()[source]

Restart or reload service and check port based on configuration.

rollback()[source]

Performs a rollback to the last deployed revision.

Rollback looks for a .done symlink that points to the revision directory for the last successful deployment. If this link doesn’t exist, a rollback isn’t possible. If it does exist, the current revision directory is replaced with the target of the link and the promote and finalize stages are re-run.

class scap.deploy.DeployLog(exe_name)[source]

Tail/filter/output events from the deploy logs.

examples:

deploy-log -v
deploy-log 'host == scap-target-01'
deploy-log 'msg ~ "some important (message|msg)"'
deploy-log 'levelno >= WARNING host == scap-target-*'
_setup_loggers()[source]

Setup logging.

main(*extra_args)[source]

Main business logic of the application.

Parsed command line arguments are available in self.arguments. Global configuration is available in self.config. Unparsed command line arguments are passed as positional arguments.

Returns:exit status
class scap.deploy.DeployMediaWiki(exe_name)[source]

Deploy mediawiki via scap3.

Ideally, this class and command will be fully merged with the scap deploy command by the end of the transition; however, there may also be unique reasons, particularly on the deployment masters, to keep this command

main(*extra_args)[source]

Run deploy-mediawiki.

scap.git

Helpers for git operations and interacting with .git directories

scap.git.add_all(location, message='Update')[source]

Add everything to repo at location as user.

scap.git.checkout(location, rev)[source]

Checkout a git repo sha at a location

scap.git.clean_tags(location, max_tags)[source]

Make sure there aren’t more than max_tags.

scap.git.default_ignore(location)[source]

Create a default .gitignore file.

scap.git.describe(location)[source]

Returns a convenient label for the current state of the git repo.

scap.git.ensure_dir(location)[source]

Ensure that we’re in a git directory. If not, explode

scap.git.fat_init(location)[source]

Initializes the given directory for git-fat use.

scap.git.fat_isinitialized(location)[source]

Returns whether git-fat has been initialized for the given directory.

scap.git.fetch(location, repo, reference=None, dissociate=True, recurse_submodules=False, shallow=False, bare=False, config=None)[source]

Fetch a git repo to a location

scap.git.garbage_collect(location)[source]

Clean up a repo.

scap.git.get_disclosable_head(repo_directory, remote_thing)[source]

Get the SHA1 of the most recent commit that can be publicly disclosed. If a commit only exists locally, it is considered private. This function will try to get the tip of the remote tracking branch, and fall back to the common ancestor of HEAD and the remote version of the local branch we’re ostensibly tracking.

Parameters:
  • repo_directory – Directory to look into
  • remote_thing – If you’re not actively tracking a remote branch, you need to provide something remote for this function to look for a common ancestor with. Otherwise, this function has no way of knowing what common tree you could possibly care about. This could be a branch, a tag, or a plain sha1
Returns:

str

scap.git.info(directory)[source]

Compute git version information for a given directory that is compatible with MediaWiki’s GitInfo class.

Parameters:directory – Directory to scan for git information
Returns:Dict of information about current repository state
scap.git.info_filename(directory, install_path, cache_path)[source]

Compute the path for a git_info cache file related to a given directory.

>>> info_filename('foo', 'foo', '')
'info.json'
>>> info_filename('foo/bar/baz', 'foo', 'xyzzy')
'xyzzy/info-bar-baz.json'
scap.git.is_dir(path)[source]

Checks if path is a git, doesn’t count submodule directories

scap.git.largefile_pull(location, implementor)[source]

Syncs all git-fat or git-lfs objects for the given repo directory.

Parameters:
  • location – Repository to work in
  • implementor – What implementation to pull with (git-lfs, git-fat)
scap.git.last_deploy_tag(location)[source]

Finds the last tag to use for this deployment

scap.git.lfs_install(*args)[source]

Run git-lfs-install with provided arguments.

If no args are provided, defaults to git lfs install –global

scap.git.list_submodules(repo)[source]

List all of the submodules of a given respository

scap.git.next_deploy_tag(location)[source]

Calculates the scap/sync/{date}/{n} tag to use for this deployment

scap.git.reflog(repo, fmt='oneline', branch=None)[source]

Fetch reflog as list

scap.git.remap_submodules(location, server)[source]

Remap all submodules to new server (tin)

This function supports remapping submodules available on the deployment server. Since the remote is a non-bare repo (as of git 1.7.8) all submodules should be available over http under the remote server checkout’s git dir: [server]/[repo]/.git/modules/[submodule_path]

Parameters:
  • location – String path to local git checkout containing a .gitmodules file
  • server – String path to remote, non-bare, repo gitdir
scap.git.remote_exists(location, remote)[source]

Check if remote exists in location

scap.git.remote_set(location, repo, remote='origin')[source]

set the remote at location to repo

scap.git.remove_all_ignores(location)[source]

Remove .gitignore files under a location.

scap.git.resolve_gitdir(directory)[source]

Find the .git directory for a given path.

This will resolve the gitlink if path/.git is a gitlink to a bare repo e.g. a file with one line, like this:

gitdir:/path/to/somewhere/else

scap.git.sha(location, rev)[source]

Returns SHA1 for things like HEAD or HEAD~~

scap.git.sync_submodules(location)[source]

Sync git submodules on target machines

scap.git.tag_repo(deploy_info, location='/src')[source]

creates new tag in deploy repo

scap.git.update_deploy_head(deploy_info, location)[source]

updates .git/DEPLOY_HEAD file

Parameters:
  • deploy_info – current deploy info to write to file as YAML
  • location ((optional)) – git directory location (default cwd)
scap.git.update_server_info(*args, **kwargs)[source]

runs git update-server-info and tags submodules

scap.git.update_submodules(location, git_remote=None, use_upstream=False, reference=None)[source]

Update git submodules on target machines

scap.context

Management of deployment host/target directories and execution context.

Copyright © 2014-2017 Wikimedia Foundation and Contributors.

This file is part of Scap.

Scap is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, version 3.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program. If not, see <http://www.gnu.org/licenses/>.

class scap.context.Context(root, environment=None)[source]

Base context for either the deployment host or target.

__init__(root, environment=None)[source]

Instantiate a new context at the given root path.

Parameters:
  • root – Root directory
  • environment – Environment name used when resolving config files.
path(*relpaths)[source]

Qualify path relative to the root path.

setup()[source]

Create the root directory, use it as the root context.

class scap.context.HostContext(root, environment=None)[source]

Manage deployment host paths and execution context.

env_specific_path(*relpaths)[source]

Return path to default or environment specific file/directory.

Both the environment specific path at scap/environments/{name} and the default path at scap is searched in respective order. The first path that exists will be returned.

env_specific_paths(*relpaths)[source]

Return paths to default and environment specific files/directories.

Both the environment specific path at scap/environments/{name} and the default path at scap is searched. Paths are included in the returned list for each one that exists.

log_path(*relpaths)[source]

Qualify the given log path.

scap_path(*relpaths)[source]

Qualify path relative to the scap directory.

setup()[source]

Create the scap and log directories as necessary.

See :class:Context.setup for its additional operations.

class scap.context.TargetContext(root, environment=None)[source]

Manage target host paths and execution context.

cache_dir

Path to the cached repo clone.

Symlink that points to the currently deployed revision.

current_rev_dir

Real path to the currently deployed revision.

done_rev_dir

Real path to the revision previously marked as done.

find_old_rev_dirs(cache_revs=5)[source]

Generate revision directories that are candidates for deletion.

The cache_revs most recent revision directories and any revision directory that is current or in progress is not considered.

Create link to the given rev’s directory at the given path.

local_config

Local target file that has a copy of the last-deployed config.

mark_rev_current(rev)[source]

Change the current rev to the given one.

This state is maintained as a current symlink in the directory root that points to the relevant revs/{rev} directory.

mark_rev_done(rev)[source]

Change the state to done for the given rev.

This state is maintained as a .done symlink in the directory root that points to the relevant revs/{rev} directory.

mark_rev_in_progress(rev)[source]

Change the state to in-progress for the given rev.

This state is maintained as a .in-progress symlink in the directory root that points to the relevant revs/{rev} directory.

rev_done

The rev that is currently marked as done.

rev_in_progress

The rev that is currently marked as in-progress.

rev_path(rev, *paths)[source]

Return the path to the given repo revision.

revs_dir

Context directory that stores revisions.

scripts_dir(rev)[source]

Path to scripts for a given rev.

setup()[source]

Create the cache and revs directory.

See :class:Context.setup for its additional operations.

General Utilities

scap.utils

Contains misc utility functions.

class scap.utils.VarDumpJSONEncoder(skipkeys=False, ensure_ascii=True, check_circular=True, allow_nan=True, sort_keys=False, indent=None, separators=None, encoding='utf-8', default=None)[source]

encode python objects to json

default(o)[source]

Implement this method in a subclass such that it returns a serializable object for o, or calls the base implementation (to raise a TypeError).

For example, to support arbitrary iterators, you could implement default like this:

def default(self, o):
    try:
        iterable = iter(o)
    except TypeError:
        pass
    else:
        return list(iterable)
    # Let the base class default method raise the TypeError
    return JSONEncoder.default(self, o)
scap.utils.ask(question, default, choices=None)[source]

Provide a y/n prompt if the controlling terminal is interactive.

Parameters:
  • question – Prompt message to display
  • default – Default answer to use in the case of a non-interactive terminal
  • choices – Optional choices to present after the question. Defaults to the default answer
Returns:

str User input or default value

scap.utils.cd(*args, **kwds)[source]

Context manager. Cds to dirname.

It moves back to previous dir on context exit. :param dirname: directory into which it should change

scap.utils.confirm(question='Continue?', default=False, on_fulfilled=None, on_rejected=None)[source]

Ask for confirmation from the user if possible, otherwise return default when stdin is not attached to a terminal.

The confirmation is fullfilled when the user types an affirming response which can be either ‘y’ or ‘yes’, otherwise the default choice is assumed

The confirmation is rejected when default=False and the user types anything other than affirmative.

Parameters:
  • question – prompt text to show to the user
  • default – boolean default choice, True [Y/n] or False [y/N]. This is the value that is returned when a tty is not attached to stdin or the user presses enter without typing a response.
  • on_fullfilled – optional callback function which is called before returning True
  • on_rejected – optional, either a callback function or an exception to be raised when we fail to get confirmation. This can be used to let the user bail out of a workflow or to bail when execution is not attached to a terminal.
scap.utils.context_logger(*args, **kwds)[source]

Context manager that maintains nested logger contexts.

Each time you enter a with block using this context manager, a named logger is set up as a child of the current logger. When exiting the with block, the logger gets popped off the stack and the parent logger takes it’s place as the ‘current’ logging context.

The easiest way to use this is to decorate a function with log_context, For Example:

@log_context('name')
def my_func(some, args, logger=None):
    logger.debug('something')
scap.utils.cpus_for_jobs()[source]

Get how many CPUs we can use for farming jobs out

scap.utils.eintr_retry(func, *args)[source]

Retry a system call if it is interrupted by EINTR.

Extracted from stdlib’s subprocess (where it is called _eintr_retry_call – the leading underscore indicating it is not part of the module’s API). This is not needed on Python >= 3.5, thanks to PEP 0475.

See <https://www.python.org/dev/peps/pep-0475/>.

scap.utils.find_nearest_host(hosts, port=22, timeout=1)[source]

Given a collection of hosts, find the one that is the fewest number of hops away.

>>> # Begin test fixture
>>> import socket
>>> fixture_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
>>> fixture_socket.bind(('127.0.0.1', 0))
>>> fixture_socket.listen(1)
>>> fixture_port = fixture_socket.getsockname()[1]
>>> # End test fixture
>>> find_nearest_host(['127.0.0.1'], port=fixture_port)
'127.0.0.1'
Parameters:
  • hosts – Hosts to check
  • port – Port to try to connect on (default: 22)
  • timeout – Timeout in seconds (default: 1)
scap.utils.find_upwards(name, starting_point='/src')[source]

Search the specified directory, and all parent directories, for a given filename, returning the first matching path that is found.

Parameters:
  • name – the relative path name to search for
  • starting_point – the directory to start searching from. Each parent directory will be searched until a match is found.
Returns:

if a match is found, returns the full path to the matching file, None otherwise.

scap.utils.get_active_wikiversions(directory, realm, datacenter)[source]

Get an ordered collection of active MediaWiki versions.

Returns:collections.OrderedDict of {version:wikidb} values sorted by version number in ascending order
scap.utils.get_env_specific_filename(path, env=None)[source]

Find a file specific to the environment in which scap is running.

scap.utils.get_patches(sub_dirs, root_dir)[source]

Find all patches under each subdirectory.

:param sub_dirs list of sub directories under which to search :param root_dir base path under which subdirectories reside :return dictionary of patches, keyed by sub_dir

scap.utils.get_real_user_fullname()[source]

Return the first entry in GECOS field for real user.

scap.utils.get_real_username()[source]

Get the username of the real user.

scap.utils.get_realm_specific_filename(filename, realm, datacenter)[source]

Find the most specific file for the given realm and datacenter.

The extension is separated from the filename and then recombined with the realm and datacenter: - base-realm-datacenter.ext - base-realm.ext - base-datacenter.ext

scap.utils.get_user_fullname(name=None)[source]

Return the first entry in GECOS field for name.

scap.utils.get_username(user=None)[source]

Get the username of the effective user.

scap.utils.human_duration(elapsed)[source]

Format an elapsed seconds count as human readable duration.

>>> human_duration(1)
'00m 01s'
>>> human_duration(65)
'01m 05s'
>>> human_duration(60*30+11)
'30m 11s'
scap.utils.is_initsystem(to_test)[source]

Check if the init system on is the one to test.

Parameters:to_test – init system to test
Returns:boolean
scap.utils.is_service_running(service)[source]

Check if a service is running.

Parameters:service – Service name
scap.utils.isclose(a, b, rel_tol=1e-09, abs_tol=0.0)[source]

Return True if a is close in value to b. False otherwise.

Parameters:
  • a – one of the values to be tested
  • b – the other value to be tested
  • rel_tol=1e-9 – The relative tolerance – the amount of error allowed, relative to the absolute value of the larger input values.
  • abs_tol=0.0 – The minimum absolute tolerance level – useful for comparisons to zero.

Copyright: Christopher H. Barker Original License: Apache License 2.0 <https://github.com/PythonCHB/close_pep>

scap.utils.iterate_subdirectories(root)[source]

Generator over the child directories of a given directory.

scap.utils.join_path(*fragments)[source]

Join several path fragments into a complete, normalized path string.

Strips leading and trailing slashes from path fragments to avoid an unfortunate feature of os.path.join() which is described in the python documentation for os.path as follows:

“If any component is an absolute path, all previous components are thrown away, and joining continues.”

scap.utils.log_context(context_name)[source]

Decorator to wrap the a function in a new context_logger.

The logger is passed to the function via a kwarg named ‘logger’.

scap.utils.make_sudo_check_call_env(env)[source]

Returns a string of environment variables formatted for the shell

sudo 1.8.21 adds support for adding a list of variables to –preserve-env, that should replace this function in future

scap.utils.md5_file(path)[source]

Compute the md5 checksum of a file’s contents.

Parameters:path – Path to file
Returns:hexdigest of md5 checksum
scap.utils.mkdir_p(path)[source]

Create directory path.

Parameters:path – The directory path to be created.
scap.utils.ordered_load(stream, Loader=<class 'yaml.loader.Loader'>, object_pairs_hook=<class 'collections.OrderedDict'>)[source]

Load yaml files and keeping order.

From stackoverflow.com/questions/5121931

:param stream the file object to read :param loader yaml.Load or its subclasses :object_pairs_hook type of return :return OrderedDict object with the same order of the yaml file

scap.utils.service_exists(service)[source]

Determine if service exists.

scap.utils.sudo_check_call(*args, **kwargs)[source]

Run a command as a specific user.

Reports stdout/stderr of process to logger during execution.

Parameters:
  • user – User to run command as
  • cmd – Command to execute
  • logger – Logger to send process output to
Raises:

subprocess.CalledProcessError on non-zero process exit

scap.utils.sudo_temp_dir(*args, **kwds)[source]

Create a temporary directory and delete it after the block.

Parameters:
  • owner – Directory owner
  • prefix – Temp directory prefix
Returns:

Full path to temporary directory

scap.utils.systemd_service_exists(service)[source]

Systemd service unit exists

scap.utils.sysv_service_exists(service)[source]

Determine if a sysvinit script exists for a service.

scap.utils.upstart_service_exists(service)[source]

Upstart service exists

scap.utils.var_dump(*args, **kwargs)[source]

dump an object to the console as pretty-printed json

scap.ansi

ANSI escape codes

..seealso:: https://en.wikipedia.org/wiki/ANSI_escape_code

Copyright © 2014-2017 Wikimedia Foundation and Contributors.

This file is part of Scap.

Scap is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, version 3.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program. If not, see <http://www.gnu.org/licenses/>.

scap.ansi.esc(*args)[source]

Get an ANSI escape code.

>>> esc(BG_WHITE, FG_RED, BLINK) == r''
True
Parameters:args – ANSI attributes
Returns:str
scap.ansi.format_ansi(*args)[source]

create an ansi color string from a list of color codes and plain strings.

>>> format_ansi((FG_BLUE,BG_WHITE),'blue on white')                  == 'blue on white'
True
Parameters:args – ANSI color codes and strings of text
Returns:str

Get the scap logo.

Scappy the scap pig:

        ___ ____
      ⎛   ⎛ ,----
       \  //==--'
  _//| .·//==--'    ____________________________
 _OO≣=-  ︶ ᴹw ⎞_§ ______  ___\ ___\ ,\__ \/ __ \
(∞)_, )  (     |  ______/__  \/ /__ / /_/ / /_/ /
  ¨--¨|| |- (  / _______\____/\___/ \__^_/  .__/
      ««_/  «_/ jgs/bd808               /_/

Ascii art derived from original work by Joan Stark [1] and the speed figlet font [2].

Parameters:
  • color – Color logo using ANSI escapes
  • colors – Alternate colors
Returns:

str

[1]http://www.oocities.org/spunk1111/farm.htm#pig
[2]http://www.jave.de/figlet/fonts/details/speed.html
scap.ansi.reset()[source]

Get the ANSI reset code.

>>> reset() == r''
True
Returns:str

scap.template

Module for working with file templates

Copyright © 2014-2017 Wikimedia Foundation and Contributors.

This file is part of Scap.

Scap is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, version 3.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program. If not, see <http://www.gnu.org/licenses/>.

class scap.template.Template(name, loader, erb_syntax=False, var_file=None, overrides=None, output_format=None)[source]

Adapter class that wraps jinja2 templates.

__init__(name, loader, erb_syntax=False, var_file=None, overrides=None, output_format=None)[source]

x.__init__(…) initializes x; see help(type(x)) for signature

_get_file_vars()[source]

Load yaml var file if it exists.

Returns:dict variables for template use
_make_env_args(loader, erb_syntax, output_format)[source]

Generate properties to pass to the jinja template.

render()[source]

Renders the templates specified by self.name.

It uses the variables sourced from the import yaml file specified by self.var_file

scap.template.get_output_formatter(fmt)[source]

Get output formatter based on desired output format.

scap.template.guess_format(fn)[source]

Guess the output format based on config file extension.

scap.template.yaml_finalize(value)[source]

Output yaml values rather than pythonic values

Logging and Monitoring

scap.log

Helpers for routing and formatting log data.

Copyright © 2014-2017 Wikimedia Foundation and Contributors.

This file is part of Scap.

Scap is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, version 3.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program. If not, see <http://www.gnu.org/licenses/>.

class scap.log.AnsiColorFormatter(fmt=None, datefmt=None, colors=None)[source]

Colorize output according to logging level.

__init__(fmt=None, datefmt=None, colors=None)[source]
Parameters:
  • fmt – Message format string
  • datefmt – Time format string
  • colors – Dict of {‘levelname’: ANSI SGR parameters}
format(record)[source]

Format the specified record as text.

The record’s attribute dictionary is used as the operand to a string formatting operation which yields the returned string. Before formatting the dictionary, a couple of preparatory steps are carried out. The message attribute of the record is computed using LogRecord.getMessage(). If the formatting string uses the time (as determined by a call to usesTime(), formatTime() is called to format the event time. If there is exception information, it is formatted using formatException() and appended to the message.

class scap.log.DeployLogFormatter(fmt=None, datefmt=None)[source]

Ensure that all deploy.log records contain a host attribute.

format(record)[source]

Format the specified record as text.

The record’s attribute dictionary is used as the operand to a string formatting operation which yields the returned string. Before formatting the dictionary, a couple of preparatory steps are carried out. The message attribute of the record is computed using LogRecord.getMessage(). If the formatting string uses the time (as determined by a call to usesTime(), formatTime() is called to format the event time. If there is exception information, it is formatted using formatException() and appended to the message.

class scap.log.DeployLogHandler(log_file)[source]

Handler for scap/deploy.log.

__init__(log_file)[source]

Open the specified file and use it as the stream for logging.

class scap.log.DiffLogFormatter(fmt=None, datefmt=None, colors=None)[source]
__init__(fmt=None, datefmt=None, colors=None)[source]
Parameters:
  • fmt – Message format string
  • datefmt – Time format string
  • colors – Dict of {‘levelname’: ANSI SGR parameters}
format(record)[source]

Format the specified record as text.

The record’s attribute dictionary is used as the operand to a string formatting operation which yields the returned string. Before formatting the dictionary, a couple of preparatory steps are carried out. The message attribute of the record is computed using LogRecord.getMessage(). If the formatting string uses the time (as determined by a call to usesTime(), formatTime() is called to format the event time. If there is exception information, it is formatted using formatException() and appended to the message.

class scap.log.FancyProgressReporter(name='', expect=0, fd=<open file '<stderr>', mode 'w'>)[source]
__init__(name='', expect=0, fd=<open file '<stderr>', mode 'w'>)[source]
Parameters:
  • name – Name of command being monitored
  • expect – Number of results to expect
  • fd – File handle to write status messages to
finish()[source]

Finish tracking progress.

class scap.log.Filter(criteria, invert=True)[source]

Generic log filter that matches record attributes against criteria.

You can provide either a glob pattern, regular expression, or lambda as each attribute criterion, and invert the logic by passing filter=False.

Examples:

Filter({'name': '*.target.*', 'host': 'scap-target-01'})
Filter({'msg': re.compile('some annoying (message|msg)')})
Filter({'levelno': lambda lvl: lvl < logging.WARNING})
Filter({'name': '*.target.*'}, invert=False)

Equivalent DSL examples:

Filter.loads('name == *.target.* host == scap-target-01')
Filter.loads('msg ~ "some annoying (message|msg)"')
Filter.loads('levelno < WARNING')f
Filter.loads('name == *.target.*', invert=False)
__init__(criteria, invert=True)[source]

x.__init__(…) initializes x; see help(type(x)) for signature

append(criteria)[source]

Append the filter with the given criteria.

Parameters:criteria (iter) – Filter criteria
filter(record)[source]

Perform filtering on a given log record.

Parameters:record (LogRecord) – Log record.
isfiltering(attribute)[source]

Whether the filter has criteria for the given attribute.

static loads(expression, invert=True)[source]

Construct a Filter from the given free-form expression.

See Filter for examples.

static parse(expression)[source]

Parse the given filter expression and generates its parts.

Parameters:expression (str) – Filter expression.
Yields:(lhs, op, rhs)
class scap.log.IRCSocketHandler(host, port, timeout=1.0)[source]

Log handler for logmsgbot on #wikimedia-operation.

Sends log events to a tcpircbot server for relay to an IRC channel.

__init__(host, port, timeout=1.0)[source]
Parameters:
  • host (str) – tcpircbot host
  • port (int) – tcpircbot listening port
  • timeout (float) – timeout for sending message
emit(record)[source]

Do whatever it takes to actually log the specified logging record.

This version is intended to be implemented by subclasses and so raises a NotImplementedError.

class scap.log.JSONFormatter(fmt=None, datefmt=None)[source]

Serialize logging output as JSON.

Can be used to maintain logged event structure between the deployment host and remote targets.

format(record)[source]

Format the specified record as text.

The record’s attribute dictionary is used as the operand to a string formatting operation which yields the returned string. Before formatting the dictionary, a couple of preparatory steps are carried out. The message attribute of the record is computed using LogRecord.getMessage(). If the formatting string uses the time (as determined by a call to usesTime(), formatTime() is called to format the event time. If there is exception information, it is formatted using formatException() and appended to the message.

class scap.log.LogstashFormatter(fmt=None, datefmt='%Y-%m-%dT%H:%M:%SZ', log_type='scap')[source]

Format log messages for logstash.

__init__(fmt=None, datefmt='%Y-%m-%dT%H:%M:%SZ', log_type='scap')[source]
Parameters:
  • fmt – Message format string (not used)
  • datefmt – Time format string
  • type – Logstash event type
converter()
gmtime([seconds]) -> (tm_year, tm_mon, tm_mday, tm_hour, tm_min,
tm_sec, tm_wday, tm_yday, tm_isdst)

Convert seconds since the Epoch to a time tuple expressing UTC (a.k.a. GMT). When ‘seconds’ is not passed in, convert the current time instead.

format(record)[source]

Format a record as a logstash v1 JSON string.

formatException(exc_info)[source]

Format the given exception as a dict.

class scap.log.MuteReporter(name='', expect=0, fd=<open file '<stderr>', mode 'w'>)[source]

A report that declines to report anything.

__init__(name='', expect=0, fd=<open file '<stderr>', mode 'w'>)[source]
Parameters:
  • name – Name of command being monitored
  • expect – Number of results to expect
  • fd – File handle to write status messages to
finish()[source]

Finish tracking progress.

class scap.log.ProgressReporter(name, expect=0, fd=<open file '<stderr>', mode 'w'>)[source]

Track and display progress of a process.

Report on the status of a multi-step process by displaying the completion percentage and succes, failure and remaining task counts on a single output line.

__init__(name, expect=0, fd=<open file '<stderr>', mode 'w'>)[source]
Parameters:
  • name – Name of command being monitored
  • expect – Number of results to expect
  • fd – File handle to write status messages to
add_failure()[source]

Record a failed task completion.

add_success()[source]

Record a sucessful task completion.

expect(count)[source]

Set expected result count.

finish()[source]

Finish tracking progress.

start()[source]

Start tracking progress.

class scap.log.Stats(**kwargs)[source]

A simple StatsD metric client.

It can log measurements and counts to a remote StatsD host. See <https://github.com/etsy/statsd/wiki/Protocol> for details.

__init__(**kwargs)[source]

x.__init__(…) initializes x; see help(type(x)) for signature

increment(name, value=1)[source]

Increment a measurement.

timing(name, milliseconds)[source]

Report a timing measurement in milliseconds.

class scap.log.Timer(**kwargs)[source]

Context manager to track and record the time taken to execute a block.

Elapsed time will be recorded to a logger and optionally a StatsD server.

>>> with Timer('example'):
...     time.sleep(0.1)
>>> s = Stats('127.0.0.1', 2003)
>>> with Timer('example', s):
...     time.sleep(0.1)

Sub-interval times can also be recorded using the mark() method.

>>> with Timer('file copy') as t:
...     time.sleep(0.1)
...     x = t.mark('copy phase 1')
...     time.sleep(0.1)
...     y = t.mark('copy phase 2')
__enter__()[source]

Enter the runtime context.

Returns:self
__exit__(exc_type, exc_value, traceback)[source]

Exit the runtime context.

__init__(**kwargs)[source]
Parameters:
  • label (str) – Label for block (e.g. ‘scap’ or ‘rsync’)
  • stats (scap.log.Stats) – StatsD client to record block invocation and duration
_record_elapsed(label, elapsed)[source]

Log the elapsed duration.

Parameters:
  • label (str) – Label for elapsed time
  • elapsed (float) – Elapsed duration
mark(label)[source]

Log the interval elapsed since the last mark call.

Parameters:label (str) – Label for block (e.g. ‘scap’ or ‘rsync’)
class scap.log.Udp2LogHandler(host, port, prefix='scap')[source]

Log handler for udp2log.

__init__(host, port, prefix='scap')[source]
Parameters:
  • host – Hostname or ip address
  • port – Port
  • prefix – Line prefix (udp2log destination)
makePickle(record)[source]

Format record as a udp2log packet.

>>> Udp2LogHandler('127.0.0.1', 12345).makePickle(
...     logging.makeLogRecord({'msg':'line1\nline2'}))
'scap line1\nscap line2\n'
>>> Udp2LogHandler('127.0.0.1', 12345).makePickle(
...     logging.makeLogRecord({'msg':'%s12'% ('0'*65500)}))
...     # doctest: +ELLIPSIS
'scap 00000...00001\n'
scap.log.reporter(message, fancy=False)[source]

Instantiate progress reporter

Message:
  • string that will be displayed to user
Fancy:
  • boolean that determines the progress bar type
scap.log.setup_loggers(cfg, console_level=20, handlers=None)[source]

Setup the logging system.

Parameters:
  • cfg – Dict of global configuration values
  • console_level – Logging level for the local console appender
  • handlers – Additional handlers

scap.checks

Deployment checks.

Definitions are typically loaded from YAML of the following format:

checks:
some_unique_check_name:
type: command command: /usr/local/bin/my_special_check stage: promote
some_other_check_name:
type: nrpe command: some_parsed_nrpe_command_name stage: promote

Copyright © 2014-2017 Wikimedia Foundation and Contributors.

This file is part of Scap.

Scap is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, version 3.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program. If not, see <http://www.gnu.org/licenses/>.

class scap.checks.Check(name, stage, environment=None, group=None, timeout=30.0, command='', **opts)[source]

Represent a loaded ‘command’ check.

Parameters:
  • name – check name
  • stage – stage after which to run the check
  • environment – environment in which to run checks
  • group – deploy group for which to run the check
  • timeout – maximum time allowed for check execution, in seconds
  • command – check command to run
__init__(name, stage, environment=None, group=None, timeout=30.0, command='', **opts)[source]

x.__init__(…) initializes x; see help(type(x)) for signature

run()[source]

Return a running CheckJob.

validate()[source]

Validate check properties.

exception scap.checks.CheckInvalid[source]
class scap.checks.CheckJob(check)[source]

Represent and control a running check.

A CheckJob begins execution immediately and should be controlled within some kind of poll loop, typically checks.execute.

__init__(check)[source]

Inititalizes a new CheckJob and begins execution.

duration()[source]

Return the current or final job duration.

isfailure()[source]

Return whether this check failed.

kill()[source]

Kill the executing process.

poll()[source]

Read output and polls the process for exit status.

If the process has exited, an (approximate) end time is recorded. This method is non-blocking typically called within an event loop like checks.execute.

timedout()[source]

Whether the job duration has exceeded the job timeout.

wait()[source]

Block for the last stdout/stderr read of the check process.

To ensure the least amount of blocking, this method should only be called within an event loop once poll has signaled that the process has exited.

class scap.checks.OverrideCheck(name, environment)[source]

Represent a loaded ‘override’ check.

__init__(name, environment)[source]

Initialize override check.

scap.checks.checktype(type_of_check)[source]

Class decorator for registering a new check type.

Parameters:type_of_check – type name
scap.checks.execute(checks, logger, concurrency=1)[source]

Execute the given checks in parallel.

Parameters:
  • checks – iterable of checks.Check objects
  • loggerlogging.Logger to send messages to
  • concurrency – level of concurrency
Returns:

tuple of the aggregate check success and list of executed checks

Return type:

(bool, list)

scap.checks.load(cfg, environment=None)[source]

Load checks from the given config dict.

Parameters:
  • cfg – config dict
  • environment – environment in which to execute checks
scap.checks.register_type(check_type, factory)[source]

Register a new check type and factory.

Parameters:
  • check_type – type name
  • factory – callable type factory

scap.nrpe

NRPE based deployment checks.

Available Icinga/NRPE command definitions must be loaded and registered for use using load or load_directory, and register, before they can be used in checks.yaml configuration.

Example checks.yaml:
checks:
service_endpoints:
type: nrpe command: check_service_endpoints stage: promote

Copyright © 2014-2017 Wikimedia Foundation and Contributors.

This file is part of Scap.

Scap is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, version 3.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program. If not, see <http://www.gnu.org/licenses/>.

class scap.nrpe.NRPECheck(name, stage, environment=None, group=None, timeout=30.0, command='', **opts)[source]

Represent a loaded ‘nrpe’ check.

validate()[source]

Validates that the configured NRPE check is available.

scap.nrpe.load(config)[source]

Load NRPE command definitions from the given configuration.

Parameters:config – NRPE configuration string
Yields:(name, command)
scap.nrpe.load_directory(config_dir)[source]

Load available local NRPE check commands from the given directory.

Parameters:config_dir – directory in which to look for NRPE configuration
Yields:(name, command)
scap.nrpe.register(commands)[source]

Register global NRPE commands for use in check configuration.

Third Party

cdblib

Imported from: python-pure-cdb
Author: David Wilson
License: MIT

Manipulate DJB’s Constant Databases. These are 2 level disk-based hash tables that efficiently handle many keys, while remaining space-efficient.

When generated databases are only used with Python code, consider using hash() rather than djb_hash() for a tidy speedup.

Note

Minor alterations made to comply with PEP8 style check and to remove attempt to import C implementation of djb_hash. – bd808, 2014-03-04

scap.cdblib.DJB_HASH(s)

Return the value of DJB’s hash function for the given 8-bit string.

>>> py_djb_hash('')
5381
>>> py_djb_hash('')
177572
>>> py_djb_hash('€')
193278953
scap.cdblib.READ_2_LE4()

S.unpack(str) -> (v1, v2, …)

Return tuple containing values unpacked according to this Struct’s format. Requires len(str) == self.size. See struct.__doc__ for more on format strings.

class scap.cdblib.Reader(data, hashfn=<function py_djb_hash>)[source]

A dictionary-like object for reading a Constant Database.

Reader accesses through a string or string-like sequence such as mmap.mmap().

__init__(data, hashfn=<function py_djb_hash>)[source]

Create an instance reading from a sequence and hash keys using hashfn.

>>> Reader(data='')
Traceback (most recent call last):
...
IOError: CDB too small
>>> Reader(data='a' * 2048) #doctest: +ELLIPSIS
<scap.cdblib.Reader object at 0x...>
items()[source]

Like dict.items().

iteritems()[source]

Like dict.iteritems(). Items are returned in insertion order.

scap.cdblib.WRITE_2_LE4()

S.pack(v1, v2, …) -> string

Return a string containing values v1, v2, … packed according to this Struct’s format. See struct.__doc__ for more on format strings.

class scap.cdblib.Writer(fp, hashfn=<function py_djb_hash>)[source]

Object for building new Constant Databases, and writing them to a seekable file-like object.

__init__(fp, hashfn=<function py_djb_hash>)[source]

Create an instance writing to a file-like object and hash keys.

It uses hashfn to hash keys.

>>> import tempfile
>>> temp_fp = tempfile.TemporaryFile()
>>> Writer(fp=temp_fp, hashfn=py_djb_hash) #doctest: +ELLIPSIS
<scap.cdblib.Writer object at 0x...>
finalize()[source]

Write the final hash tables to the output file, and write out its index. The output file remains open upon return.

put(key, value='')[source]

Write a string key/value pair to the output file.

scap.cdblib.py_djb_hash(s)[source]

Return the value of DJB’s hash function for the given 8-bit string.

>>> py_djb_hash('')
5381
>>> py_djb_hash('')
177572
>>> py_djb_hash('€')
193278953