Scap

        ___ ____
      ⎛   ⎛ ,----
       \  //==--'
  _//| .·//==--'    ____________________________
 _OO≣=-  ︶ ᴹw ⎞_§ ______  ___\ ___\ ,\__ \/ __ \
(∞)_, )  (     |  ______/__  \/ /__ / /_/ / /_/ /
  ¨--¨|| |- (  / _______\___/ \___/ \__^_/  .__/
      ««_/  «_/ jgs/bd808               /_/

Scap is the deployment script used by Wikimedia Foundation to publish code and configuration on production web servers.

Running tests

Automated tests can be executed with make test.

Scap comes with an automated suite invoked via scripts/check. The script does not install any dependencies though.

For a list of Scap dependencies during testing, see requirements.txt and test-requirements.txt. Some Debian packages are also required to set up the running environment, for instance to install Python or pip.

To ensure compatiblity with multiple Debian distributions, we use Docker images defined using Blubber (see .pipeline/blubber.yaml). The image build installs the runtime dependencies from Debian packages while tests dependencies defined in test-requirements.txt are installed via pip. This lets us use ensure we run with the same set of Python module provided by Debian while using more recent versions of testing utilities than the one frozen by Debian (ex: pytest) and ensure we use the same version of flake8 regardless of the Debian distribution version.

To generate the container images and run tests in each of them, we provide a Makefile. To run all tests against all supported Debian distributions use make test.

The test target builds the images using the Blubberfile syntax <https://wikitech.wikimedia.org/wiki/Blubber/User_Guide#Blubberfiles> using Blubber buildkit <docker-registry.wikimedia.org/wikimedia/blubber-buildkit>. The target then invokes the defined entrypoint scripts/check which runs the tests in each images.

The Wikimedia CI builds those images in a similar way and run the same tests.

To build all images without running tests, use make images, useful when amending the Blubber config or changing dependencies.

For each Debian distribution defined at the top of the makefile, rules let you build and test against a single distribution: make test-bullseye. You will probably want to use it when developing instead of testing against each of the distribution. When your change is ready, run it against all distributions to catch potential compatibility issues (make test).

Reporting Issues

You can report issues to the #scap project on phabricator.wikimedia.org

Credits

ASCII art derived from original work by Joan Stark [1] and the speed figlet font [2].

License

Copyright 2014-2022 Wikimedia Foundation & Contributors.

Scap is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, version 3.

Scap is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public License
along with this program. If not, see <http://www.gnu.org/licenses/>.

Background

Scap is a tool that was originally designed to deploy MediaWiki code on Wikimedia Foundation servers. In Summer/Fall 2015 functionality was added to Scap to allow it to deploy any git-based repositories on our deployment server.

What does “Scap” mean?

In about 2004, in the course of moving the wiki document roots from NFS to the local hard drives of the Apache servers, a number of shell scripts were introduced, probably written by Brion, notably:

  • sync-common, which pulled the shared code into the local server

  • sync-common-all, which ran sync-common on all servers

Tim added a script which would push the MW source code in common/php out to all servers, in addition to running a pre-deployment lint check, in order to reduce the (then frequent) incidents of site downtime due to PHP fatal errors. The natural name for this script, following the previous convention, would have been sync-common-php-all, but this had two problems:

  • It was too long

  • Its acronym was not pronounceable

Tim noticed that if the last two components of the name were swapped, a short and catchy acronym could be contrived. Thus, Scap was born.

Indices and tables

Table of contents

Topic Index