Sync Commands

These commands, for the most part, use rsync to synchronize the state of the mediawiki config and wmf/* code branches from the deploy server to the production web servers. There are also commands to perform other related house-keeping tasks as described in the individual command descriptions.

scap sync

scap sync is the driver script for syncing the MediaWiki versions and configuration files currently staged on the deploy server to the rest of the cluster.

usage: scap sync [-h] [--conf CONF_FILE] [--no-shared-authsock]
                 [-D <name>:<value>] [-v] [--environment ENVIRONMENT]
                 [--no-log-message] [--force]
                 [message [message ...]]

Deploy MediaWiki to the cluster.

positional arguments:
  message               Log message for SAL

optional arguments:
  -h, --help            show this help message and exit
  --force               Skip canary checks

global arguments:
  Although these arguments can be passed to all scap (sub-)commands,         
  not all commands are affected by every global argument.

  --conf CONF_FILE      Path to configuration file
  --no-shared-authsock  Ignore any shared ssh-auth configuration
  -D <name>:<value>, --define <name>:<value>
                        Set a configuration value
  -v, --verbose         Verbose output
  --environment ENVIRONMENT
                        environment in which to execute scap
  --no-log-message      Do not log messages to the public (IRC)

#. Validate php syntax of wmf-config and multiversion
#. Sync deploy directory on localhost with staging area
#. Create/update git repo in staging area
#. Compile wikiversions.json to php in deploy directory
#. Update l10n files in staging area
#. Compute git version information
#. Commit all changes to local git repo in deploy directory
#. Ask scap masters to sync with current master
#. Ask scap proxies to sync with master server
#. Ask apaches to sync with fastest rsync server
#. Ask apaches to rebuild l10n CDB files
#. Update wikiversions.php on localhost
#. Ask apaches to sync wikiversions.php
#. Run refreshMessageBlobs.php
#. Rolling invalidation of all opcache for php 7.x

scap pull

scap pull uses rsync to fetch MediaWiki code and configuration to the local host. It is typically called automatically on hosts during the execution of scap sync.

usage: scap pull [-h] [--conf CONF_FILE] [--no-shared-authsock]
                 [-D <name>:<value>] [-v] [--environment ENVIRONMENT]
                 [--no-log-message] [--no-update-l10n] [-i INCLUDE]
                 [--delete-excluded]
                 ...

Sync local MediaWiki deployment directory with deploy server state.

positional arguments:
  servers               Rsync server(s) to copy from

optional arguments:
  -h, --help            show this help message and exit
  --no-update-l10n      Do not update l10n cache files.
  -i INCLUDE, --include INCLUDE
                        Rsync include pattern to limit transfer to. End
                        directories with a trailing `/***`. Can be used
                        multiple times.
  --delete-excluded     Also delete local files not found on the master.

global arguments:
  Although these arguments can be passed to all scap (sub-)commands, not all
  commands are affected by every global argument.

  --conf CONF_FILE      Path to configuration file
  --no-shared-authsock  Ignore any shared ssh-auth configuration
  -D <name>:<value>, --define <name>:<value>
                        Set a configuration value
  -v, --verbose         Verbose output
  --environment ENVIRONMENT
                        environment in which to execute scap
  --no-log-message      Do not log messages to the public (IRC)

scap sync-file

scap sync-file synchronizes a file or directory from the staging directory to the cluster.

usage: scap sync-file [-h] [--conf CONF_FILE] [--no-shared-authsock]
                      [-D <name>:<value>] [-v] [--environment ENVIRONMENT]
                      [--no-log-message] [--force]
                      file [message [message ...]]

Sync a specific file/directory to the cluster.

positional arguments:
  file                  File/directory to sync
  message               Log message for SAL

optional arguments:
  -h, --help            show this help message and exit
  --force               Skip canary checks

global arguments:
  Although these arguments can be passed to all scap (sub-)commands, not all
  commands are affected by every global argument.

  --conf CONF_FILE      Path to configuration file
  --no-shared-authsock  Ignore any shared ssh-auth configuration
  -D <name>:<value>, --define <name>:<value>
                        Set a configuration value
  -v, --verbose         Verbose output
  --environment ENVIRONMENT
                        environment in which to execute scap
  --no-log-message      Do not log messages to the public (IRC)

See also

scap sync-l10n

scap sync-l10n synchronizes the localization files for a given MediaWiki version to the cluster and rebuilds the associated cache files.

usage: scap sync-l10n [-h] [--conf CONF_FILE] [--no-shared-authsock]
                      [-D <name>:<value>] [-v] [--environment ENVIRONMENT]
                      [--no-log-message] [--force]
                      version [message [message ...]]

Sync l10n files for a given branch and rebuild cache files.

positional arguments:
  version               MediaWiki version (eg 1.27.0-wmf.7)
  message               Log message for SAL

optional arguments:
  -h, --help            show this help message and exit
  --force               Skip canary checks

global arguments:
  Although these arguments can be passed to all scap (sub-)commands, not all
  commands are affected by every global argument.

  --conf CONF_FILE      Path to configuration file
  --no-shared-authsock  Ignore any shared ssh-auth configuration
  -D <name>:<value>, --define <name>:<value>
                        Set a configuration value
  -v, --verbose         Verbose output
  --environment ENVIRONMENT
                        environment in which to execute scap
  --no-log-message      Do not log messages to the public (IRC)

See also

scap sync-wikiversions

scap sync-wikiversions compiles wikiversions.json into a CDB database and then syncs both the JSON and CDB versions to the rest of the cluster.

usage: scap sync-wikiversions [-h] [--conf CONF_FILE] [--no-shared-authsock]
                              [-D <name>:<value>] [-v]
                              [--environment ENVIRONMENT] [--no-log-message]
                              [--force]
                              [message [message ...]]

Rebuild and sync wikiversions.php to the cluster.

positional arguments:
  message               Log message for SAL

optional arguments:
  -h, --help            show this help message and exit
  --force               Skip canary checks

global arguments:
  Although these arguments can be passed to all scap (sub-)commands, not all
  commands are affected by every global argument.

  --conf CONF_FILE      Path to configuration file
  --no-shared-authsock  Ignore any shared ssh-auth configuration
  -D <name>:<value>, --define <name>:<value>
                        Set a configuration value
  -v, --verbose         Verbose output
  --environment ENVIRONMENT
                        environment in which to execute scap
  --no-log-message      Do not log messages to the public (IRC)

Misc. Commands

scap wikiversions-inuse

scap wikiversions-inuse examines wikiversions.json to find the current active MediaWiki versions.

usage: scap wikiversions-inuse [-h] [--conf CONF_FILE] [--no-shared-authsock]
                               [-D <name>:<value>] [-v]
                               [--environment ENVIRONMENT] [--no-log-message]
                               [--withdb]

Get a list of the active MediaWiki versions.

optional arguments:
  -h, --help            show this help message and exit
  --withdb              Add `=wikidb` with some wiki using the version.

global arguments:
  Although these arguments can be passed to all scap (sub-)commands, not all
  commands are affected by every global argument.

  --conf CONF_FILE      Path to configuration file
  --no-shared-authsock  Ignore any shared ssh-auth configuration
  -D <name>:<value>, --define <name>:<value>
                        Set a configuration value
  -v, --verbose         Verbose output
  --environment ENVIRONMENT
                        environment in which to execute scap
  --no-log-message      Do not log messages to the public (IRC)

scap wikiversions-compile

wikiversions-compile compiles wikiversions.json into wikiversions.php.

usage: scap wikiversions-compile [-h] [--conf CONF_FILE]
                                 [--no-shared-authsock] [-D <name>:<value>]
                                 [-v] [--environment ENVIRONMENT]
                                 [--no-log-message]

Compile wikiversions.json to wikiversions.php.

optional arguments:
  -h, --help            show this help message and exit

global arguments:
  Although these arguments can be passed to all scap (sub-)commands, not all
  commands are affected by every global argument.

  --conf CONF_FILE      Path to configuration file
  --no-shared-authsock  Ignore any shared ssh-auth configuration
  -D <name>:<value>, --define <name>:<value>
                        Set a configuration value
  -v, --verbose         Verbose output
  --environment ENVIRONMENT
                        environment in which to execute scap
  --no-log-message      Do not log messages to the public (IRC)

scap cdb-rebuild

scap cdb-rebuild rebuilds localization cache CDB files from JSON files.

usage: scap cdb-rebuild [-h] [--conf CONF_FILE] [--no-shared-authsock]
                        [-D <name>:<value>] [-v] [--environment ENVIRONMENT]
                        [--no-log-message] [--version VERSION] [--no-progress]
                        [--staging]

Rebuild localization cache CDB files from the JSON versions.

optional arguments:
  -h, --help            show this help message and exit
  --version VERSION     MediaWiki version (eg 1.27.0-wmf.7)
  --no-progress         Do not show progress indicator.
  --staging             Rebuild cdb files in staging directory

global arguments:
  Although these arguments can be passed to all scap (sub-)commands, not all
  commands are affected by every global argument.

  --conf CONF_FILE      Path to configuration file
  --no-shared-authsock  Ignore any shared ssh-auth configuration
  -D <name>:<value>, --define <name>:<value>
                        Set a configuration value
  -v, --verbose         Verbose output
  --environment ENVIRONMENT
                        environment in which to execute scap
  --no-log-message      Do not log messages to the public (IRC)

scap cdb-json-refresh

refreshCdbJsonFiles Create JSON/MD5 files for all CDB files in a directory.

usage: scap cdb-json-refresh [-h] [--conf CONF_FILE] [--no-shared-authsock]
                             [-D <name>:<value>] [-v]
                             [--environment ENVIRONMENT] [--no-log-message] -d
                             DIRECTORY [-t THREADS]

Create JSON/MD5 files for all CDB files in a directory.

optional arguments:
  -h, --help            show this help message and exit
  -d DIRECTORY, --directory DIRECTORY
                        Directory containing cdb files
  -t THREADS, --threads THREADS
                        Number of threads to use to build json/md5 files

global arguments:
  Although these arguments can be passed to all scap (sub-)commands,         
  not all commands are affected by every global argument.

  --conf CONF_FILE      Path to configuration file
  --no-shared-authsock  Ignore any shared ssh-auth configuration
  -D <name>:<value>, --define <name>:<value>
                        Set a configuration value
  -v, --verbose         Verbose output
  --environment ENVIRONMENT
                        environment in which to execute scap
  --no-log-message      Do not log messages to the public (IRC)

This will put a JSON and MD5 file in /upstream for each CDB file.

This can be combined with rsync and the scap-rebuild-cdbs to
push out large CDB files with minimal traffic. CDB files change
drastically with small key/value changes, where as JSON files do not, and
thus they diff/rdiff much better.

When pushing updates with rsync, this should be run before running rsync.
The rsync command should exclude CDB files or at least use
-ignore-existing. After the rsync is done, scap-rebuild-cdbs can be
run on each server to apply the updates to the CDB files.

See also