Sync Commands

These commands, for the most part, use rsync to synchronize the state of the mediawiki config and wmf/* code branches from the deploy server to the production web servers. There are also commands to perform other related house-keeping tasks as described in the individual command descriptions.

scap sync-world

scap sync-world is the driver script for syncing the MediaWiki versions and configuration files currently staged on the deploy server to the rest of the cluster.

usage: scap sync-world [-h] [--conf CONF_FILE] [--no-shared-authsock]
                       [-D <name>:<value>] [--verbose]
                       [--environment ENVIRONMENT] [--no-log-message]
                       [--no-local-config] [--force] [-w <time in secs>]
                       [--skip-l10n-update] [-n] [--stop-before-sync]
                       [--no-logo] [--pause-after-testserver-sync]
                       [--notify-user NOTIFY_USER] [--k8s-only]
                       [--k8s-confirm-diffs]
                       [message [message ...]]

Deploy MediaWiki to the cluster.

positional arguments:
  message               Log message for SAL

optional arguments:
  -h, --help            show this help message and exit
  --force               Skip canary checks, performs ungraceful php-fpm
                        restarts
  -w <time in secs>, --canary-wait-time <time in secs>
                        Define how long new code will run on the canary
                        servers (default is 20s)
  --skip-l10n-update    Skip update of l10n files
  -n                    No-op for running tests
  --stop-before-sync    Perform all operations up to but not including
                        rsyncing to any host
  --no-logo             Do not print the Scap logo
  --pause-after-testserver-sync
                        Pause after syncing testservers and prompt the user to
                        confirm to continue syncing
  --notify-user NOTIFY_USER
                        User to notify on IRC after sync to testservers. Can
                        be used multiple times
  --k8s-only            Deploy/sync to Kubernetes targets only
  --k8s-confirm-diffs   Display and require confirmation of helmfile diffs
                        before proceeding.

global arguments:
  Although these arguments can be passed to all scap (sub-)commands,         
  not all commands are affected by every global argument.

  --conf CONF_FILE      Path to configuration file
  --no-shared-authsock  Ignore any shared ssh-auth configuration
  -D <name>:<value>, --define <name>:<value>
                        Set a configuration value
  --verbose             Verbose output
  --environment ENVIRONMENT
                        environment in which to execute scap
  --no-log-message      Do not log messages to the public (IRC)
  --no-local-config     Do not load configuration from the ./scap directory

#. Validate php syntax of wmf-config and multiversion
#. Compile wikiversions.json to php in staging directory
#. Update l10n files in staging area
#. Compute git version information
#. Ask scap masters to sync with current master
#. Ask scap proxies to sync with master server
#. Ask apaches to sync with fastest rsync server (excluding wikiversions.php)
#. Ask apaches to rebuild l10n CDB files
#. Ask apaches to sync wikiversions.php
#. Run purgeMessageBlobStore.php
#. Rolling invalidation of all opcache for php 7.x

See also

• scap.ScapWorld()

• scap.tasks.check_valid_syntax()

• scap.tasks.compile_wikiversions()

• scap.tasks.sync_common()

• scap.tasks.sync_wikiversions()

scap pull

scap pull uses rsync to fetch MediaWiki code and configuration to the local host. It is typically called automatically on hosts during the execution of scap sync-world.

usage: scap pull [-h] [--conf CONF_FILE] [--no-shared-authsock]
                 [-D <name>:<value>] [--verbose] [--environment ENVIRONMENT]
                 [--no-log-message] [--no-local-config] [--no-update-l10n]
                 [--exclude-wikiversions.php] [-i INCLUDE] [--delete-excluded]
                 [--no-php-restart]
                 ...

Sync local MediaWiki deployment directory with deploy server state.

positional arguments:
  servers               Rsync server(s) to copy from

optional arguments:
  -h, --help            show this help message and exit
  --no-update-l10n      Do not update l10n cache files.
  --exclude-wikiversions.php
                        Do not rsync wikiversions.php.
  -i INCLUDE, --include INCLUDE
                        Rsync include pattern to limit transfer to. End
                        directories with a trailing `/***`. Can be used
                        multiple times.
  --delete-excluded     Also delete local files not found on the master.
  --no-php-restart      Don't restart php-fpm after the pull.

global arguments:
  Although these arguments can be passed to all scap (sub-)commands, not all
  commands are affected by every global argument.

  --conf CONF_FILE      Path to configuration file
  --no-shared-authsock  Ignore any shared ssh-auth configuration
  -D <name>:<value>, --define <name>:<value>
                        Set a configuration value
  --verbose             Verbose output
  --environment ENVIRONMENT
                        environment in which to execute scap
  --no-log-message      Do not log messages to the public (IRC)
  --no-local-config     Do not load configuration from the ./scap directory

See also

• scap.SyncCommon()

• scap.tasks.sync_common()

scap sync-file

scap sync-file synchronizes a file or directory from the staging directory to the cluster.

usage: scap sync-file [-h] [--conf CONF_FILE] [--no-shared-authsock]
                      [-D <name>:<value>] [--verbose]
                      [--environment ENVIRONMENT] [--no-log-message]
                      [--no-local-config] [--force]
                      [--pause-after-testserver-sync]
                      [--notify-user NOTIFY_USER]
                      file [message [message ...]]

Sync a specific file/directory to the cluster.

positional arguments:
  file                  File/directory to sync
  message               Log message for SAL

optional arguments:
  -h, --help            show this help message and exit
  --force               Skip canary checks
  --pause-after-testserver-sync
                        Pause after syncing testservers and prompt the user to
                        confirm to continue syncing
  --notify-user NOTIFY_USER
                        User to notify on IRC after sync to testservers. Can
                        be used multiple times

global arguments:
  Although these arguments can be passed to all scap (sub-)commands, not all
  commands are affected by every global argument.

  --conf CONF_FILE      Path to configuration file
  --no-shared-authsock  Ignore any shared ssh-auth configuration
  -D <name>:<value>, --define <name>:<value>
                        Set a configuration value
  --verbose             Verbose output
  --environment ENVIRONMENT
                        environment in which to execute scap
  --no-log-message      Do not log messages to the public (IRC)
  --no-local-config     Do not load configuration from the ./scap directory

See also

• scap.SyncFile()

scap sync-wikiversions

scap sync-wikiversions compiles wikiversions.json into a CDB database and then syncs both the JSON and CDB versions to the rest of the cluster.

usage: scap sync-wikiversions [-h] [--conf CONF_FILE] [--no-shared-authsock]
                              [-D <name>:<value>] [--verbose]
                              [--environment ENVIRONMENT] [--no-log-message]
                              [--no-local-config] [--force]
                              [--stop-before-sync]
                              [--pause-after-testserver-sync]
                              [--notify-user NOTIFY_USER] [--k8s-only]
                              [--k8s-confirm-diffs]
                              [message [message ...]]

Rebuild and sync wikiversions.php to the cluster.

positional arguments:
  message               Log message for SAL

optional arguments:
  -h, --help            show this help message and exit
  --force               Skip error checks, testserver and canary checks,
                        performs ungraceful php-fpm restarts
  --stop-before-sync    Perform all operations up to but not including
                        rsyncing to any host
  --pause-after-testserver-sync
                        Pause after syncing testservers and prompt the user to
                        confirm to continue syncing
  --notify-user NOTIFY_USER
                        User to notify on IRC after sync to testservers. Can
                        be used multiple times
  --k8s-only            Deploy/sync to Kubernetes targets only
  --k8s-confirm-diffs   Display and require confirmation of helmfile diffs
                        before proceeding.

global arguments:
  Although these arguments can be passed to all scap (sub-)commands, not all
  commands are affected by every global argument.

  --conf CONF_FILE      Path to configuration file
  --no-shared-authsock  Ignore any shared ssh-auth configuration
  -D <name>:<value>, --define <name>:<value>
                        Set a configuration value
  --verbose             Verbose output
  --environment ENVIRONMENT
                        environment in which to execute scap
  --no-log-message      Do not log messages to the public (IRC)
  --no-local-config     Do not load configuration from the ./scap directory

See also

• scap.SyncWikiversions()

• scap.tasks.compile_wikiversions()

• scap.tasks.sync_wikiversions()

Misc. Commands

scap wikiversions-inuse

scap wikiversions-inuse examines wikiversions.json to find the current active MediaWiki versions.

usage: scap wikiversions-inuse [-h] [--conf CONF_FILE] [--no-shared-authsock]
                               [-D <name>:<value>] [--verbose]
                               [--environment ENVIRONMENT] [--no-log-message]
                               [--no-local-config] [--withdb] [--staging]

Get a list of the active MediaWiki versions.

optional arguments:
  -h, --help            show this help message and exit
  --withdb              Add `=wikidb` with some wiki using the version.
  --staging             Read wikiversions from the staging directory. This
                        flag is deprecated since wikiversions-inuse always
                        uses the staging directory.

global arguments:
  Although these arguments can be passed to all scap (sub-)commands, not all
  commands are affected by every global argument.

  --conf CONF_FILE      Path to configuration file
  --no-shared-authsock  Ignore any shared ssh-auth configuration
  -D <name>:<value>, --define <name>:<value>
                        Set a configuration value
  --verbose             Verbose output
  --environment ENVIRONMENT
                        environment in which to execute scap
  --no-log-message      Do not log messages to the public (IRC)
  --no-local-config     Do not load configuration from the ./scap directory

See also

• scap.MWVersionsInUse()

scap wikiversions-compile

wikiversions-compile compiles wikiversions.json into wikiversions.php.

usage: scap wikiversions-compile [-h] [--conf CONF_FILE]
                                 [--no-shared-authsock] [-D <name>:<value>]
                                 [--verbose] [--environment ENVIRONMENT]
                                 [--no-log-message] [--no-local-config]
                                 [--staging]

Compile wikiversions.json to wikiversions.php.

optional arguments:
  -h, --help            show this help message and exit
  --staging             Compile wikiversions in staging directory. This flag
                        is deprecated since wikiversions-compile always uses
                        the staging directory.

global arguments:
  Although these arguments can be passed to all scap (sub-)commands, not all
  commands are affected by every global argument.

  --conf CONF_FILE      Path to configuration file
  --no-shared-authsock  Ignore any shared ssh-auth configuration
  -D <name>:<value>, --define <name>:<value>
                        Set a configuration value
  --verbose             Verbose output
  --environment ENVIRONMENT
                        environment in which to execute scap
  --no-log-message      Do not log messages to the public (IRC)
  --no-local-config     Do not load configuration from the ./scap directory

See also

• scap.CompileWikiversions()

• scap.tasks.compile_wikiversions()

scap cdb-rebuild

scap cdb-rebuild rebuilds localization cache CDB files from JSON files.

usage: scap cdb-rebuild [-h] [--conf CONF_FILE] [--no-shared-authsock]
                        [-D <name>:<value>] [--verbose]
                        [--environment ENVIRONMENT] [--no-log-message]
                        [--no-local-config] [--version VERSION]
                        [--no-progress] [--master]

Rebuild localization cache CDB files from the JSON versions.

optional arguments:
  -h, --help            show this help message and exit
  --version VERSION     MediaWiki version (eg 1.27.0-wmf.7)
  --no-progress         Do not show progress indicator.
  --master              Operate in a mode suitable for a deploy master

global arguments:
  Although these arguments can be passed to all scap (sub-)commands, not all
  commands are affected by every global argument.

  --conf CONF_FILE      Path to configuration file
  --no-shared-authsock  Ignore any shared ssh-auth configuration
  -D <name>:<value>, --define <name>:<value>
                        Set a configuration value
  --verbose             Verbose output
  --environment ENVIRONMENT
                        environment in which to execute scap
  --no-log-message      Do not log messages to the public (IRC)
  --no-local-config     Do not load configuration from the ./scap directory

See also

• scap.RebuildCdbs()

• scap.tasks.merge_cdb_updates()

scap cdb-json-refresh

refreshCdbJsonFiles Create JSON/MD5 files for all CDB files in a directory.

usage: scap cdb-json-refresh [-h] [--conf CONF_FILE] [--no-shared-authsock]
                             [-D <name>:<value>] [--verbose]
                             [--environment ENVIRONMENT] [--no-log-message]
                             [--no-local-config] -d DIRECTORY [-t THREADS]

Create JSON/MD5 files for all CDB files in a directory.

optional arguments:
  -h, --help            show this help message and exit
  -d DIRECTORY, --directory DIRECTORY
                        Directory containing cdb files
  -t THREADS, --threads THREADS
                        Number of threads to use to build json/md5 files

global arguments:
  Although these arguments can be passed to all scap (sub-)commands,         
  not all commands are affected by every global argument.

  --conf CONF_FILE      Path to configuration file
  --no-shared-authsock  Ignore any shared ssh-auth configuration
  -D <name>:<value>, --define <name>:<value>
                        Set a configuration value
  --verbose             Verbose output
  --environment ENVIRONMENT
                        environment in which to execute scap
  --no-log-message      Do not log messages to the public (IRC)
  --no-local-config     Do not load configuration from the ./scap directory

This will put a JSON and MD5 file in /upstream for each CDB file.

This can be combined with rsync and the scap-rebuild-cdbs to
push out large CDB files with minimal traffic. CDB files change
drastically with small key/value changes, where as JSON files do not, and
thus they diff/rdiff much better.

When pushing updates with rsync, this should be run before running rsync.
The rsync command should exclude CDB files or at least use
-ignore-existing. After the rsync is done, scap-rebuild-cdbs can be
run on each server to apply the updates to the CDB files.

See also

• scap.refreshCdbJsonFiles()

• scap.tasks.refresh_cdb_json_files()