Quibble: a test runner for MediaWiki

Quibble gets MediaWiki, install it and run all MediaWiki tests suites. All is included in a single command.

As a requirement one needs python 3 and all software required by MediaWiki and its tests systems:

  • Chromium
  • composer
  • NodeJS
  • npm
  • php
  • python 3
  • Xvfb

TLDR:

docker build --tag quibble .
docker run -it --rm quibble

Which runs tests with php7.0, MariaDB and using mediawiki/vendor.git to provide PHP libraries.

You could instead run tests with dependencies provided by composer install and use SQLite as a database backend:

docker run -it quibble  --packages-source composer --db sqlite

Wikimedia maintains Docker containers intended to be used for its continuous integration system:

docker pull docker-registry.wikimedia.org/releng/quibble-stretch-php72:latest

The source is on Gerrit https://gerrit.wikimedia.org/g/integration/config under the dockerfiles directory. Other containers with slight variation such as providing Zend PHP5.5 or HHVM.

Further documentation can be found on https://doc.wikimedia.org/quibble/ .

Setup

Docker container

Get the latest image being run by Wikimedia CI:

docker pull docker-registry.wikimedia.org/releng/quibble-stretch-php72:latest

Quibble clones the repositories from Gerrit and then run composer and npm. At the end of the run, the container would be dismissed as well as all the downloaded content. To make it faster, you should set local copies of the git repositories that would be downloaded from and set up a cache directory on the host to be mounted in the container.

To avoid cloning MediaWiki over the network, you should initialize local bare git repositories to be used as a reference for git to copy them from:

mkdir -p ref/mediawiki/skins
git clone --bare https://gerrit.wikimedia.org/r/mediawiki/core ref/mediawiki/core.git
git clone --bare https://gerrit.wikimedia.org/r/mediawiki/vendor ref/mediawiki/vendor.git
git clone --bare https://gerrit.wikimedia.org/r/mediawiki/skins/Vector ref/mediawiki/skins/Vector.git

The Docker containers have XDG_CACHE_HOME=/cache set which is recognized by package managers. Create a cache directory writable by any user:

mkdir cache
chmod 777 cache

Commands write logs into /workspace/log, you can create one on the host and mount it in the container:

mkdir -p log
chmod 777 log

You might also want to reuse the installed sources between runs. The container has the source repository under /workspace/src:

mkdir -p src
chmod 777 src

The directory tree on the host will looks like:

.
├── cache/
├── log/
├── src/
└── ref/
    └── mediawiki/
        ├── core.git/
        ├── skins/
        │   └── Vector.git/
        └── vendor.git/

When running the Docker container, mount the directories from the host:

Host dir Container dir Docker run argument
./cache/ /cache -v "$(pwd)"/cache:/cache
./log/ /workspace/log -v "$(pwd)"/log:/workspace/log
./ref/ /srv/git -v "$(pwd)"/ref:/srv/git:ro
./src/ /workspace/src -v "$(pwd)"/src:/workspace/src

The final command:

docker run -it --rm \
  -v "$(pwd)"/cache:/cache \
  -v "$(pwd)"/log:/workspace/log \
  -v "$(pwd)"/ref:/srv/git:ro \
  -v "$(pwd)"/src:/workspace/src \
  docker-registry.wikimedia.org/releng/quibble-stretch-php72:latest

Quibble will then do the initial cloning of repositories reusing bare repositories from ref, being local it is arguably faster than transferring everything from Gerrit. The composer install and npm install will save the downloaded packages to cache which speed up the next run.

Finally, having /src mounted from the host, lets one reuse the installed wiki. One can later skip cloning/checking out the repositories by passing --skip-zuul and skip installing composer and npm dependencies with --skip-deps. For other options see: Quibble CLI usage.

TESTING

Coverage report:

tox -e cover && open cover/index.html

LICENSE

Files under zuul comes from Zuul “A Project Gating System”:

Copyright 2012 Hewlett-Packard Development Company, L.P. Copyright 2013-2014 OpenStack Foundation Copyright 2013-2018 Antoine Musso Copyright 2014-2018 Wikimedia Foundation Inc. Copyright 2015 Rackspace Australia

quibble/gitchangedinhead.py comes from Wikimedia CI scripts:

Copyright 2013, 2018, Antoine Musso Copyright 2017, Kunal Mehta Copyright 2017, 2018, Wikimedia Foundation Inc.

Other files are:

Copyright 2017-2018, Antoine Musso Copyright 2017, Tyler Cipriani

Licensed under the Apache License, Version 2.0 (the “License”); you may not use this file except in compliance with the License. You may obtain a copy of the License at

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an “AS IS” BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Reproducing a CI build

Quibble allows you to locally replicate test runs as you witnessed them in CI. This can be useful when debugging integration with projects you are not normally involved with and don’t keep a local copy of, or other obscure problems.

Create a .env file to specify the variables needed to replicate a CI run.

You can find them in the [Jenkins job parameters](https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-hhvm-docker/9262/parameters/). E.g.:

EXT_DEPENDENCIES=mediawiki/extensions/BetaFeatures\nmediawiki/extensions/Capiunto\nmediawiki/extensions/CentralAuth\nmediawiki/extensions/CirrusSearch\nmediawiki/extensions/Cite\nmediawiki/extensions/Echo\nmediawiki/extensions/EducationProgram\nmediawiki/extensions/Elastica\nmediawiki/extensions/EventLogging\nmediawiki/extensions/GeoData\nmediawiki/extensions/GuidedTour\nmediawiki/extensions/PdfHandler\nmediawiki/extensions/PropertySuggester\nmediawiki/extensions/Scribunto\nmediawiki/extensions/SiteMatrix\nmediawiki/extensions/SyntaxHighlight_GeSHi\nmediawiki/extensions/TimedMediaHandler\nmediawiki/extensions/UniversalLanguageSelector\nmediawiki/extensions/VisualEditor\nmediawiki/extensions/WikiEditor\nmediawiki/extensions/Wikibase\nmediawiki/extensions/WikibaseLexeme\nmediawiki/extensions/WikibaseQuality\nmediawiki/extensions/WikibaseQualityConstraints\nmediawiki/extensions/WikimediaBadges\nmediawiki/extensions/cldr
ZUUL_CHANGE=449454
ZUUL_CHANGE_IDS=449453,1 449454,1
ZUUL_CHANGES=mediawiki/extensions/WikibaseLexeme:master:refs/changes/53/449453/1^mediawiki/extensions/ContentTranslation:master:refs/changes/54/449454/1
ZUUL_PATCHSET=1
ZUUL_REF=refs/zuul/master/Za65b9a0ba15a4c9cad4a8a60a6357f5f
ZUUL_COMMIT=af14dcbcc39b58d9ae8d06fef6bb0ee997711a3f
ZUUL_URL=git://contint2001.wikimedia.org
ZUUL_PROJECT=mediawiki/extensions/ContentTranslation
MW_COMPOSER_MERGE_MW_IN_VENDOR=1
ZUUL_BRANCH=master
PHP_BIN=hhvm

In this example we are testing the integration of many mediawiki extensions, in particular with a change in WikibaseLexeme having an adverse effect on a change in ContentTranslation.

Not all of the variables visible in the Jenkins jobs parameters are needed. The important ones are:

EXT_DEPENDENCIES
ZUUL_PROJECT
ZUUL_REF
ZUUL_BRANCH

At the moment only builds with one change set can be reproduced locally because git://contint2001.wikimedia.org is not accessible remotely.

Choose the right docker image.

You must also choose the correct quibble image for the base OS and php interpreter to mirror the job: e.g. Debian Stretch and hhvm:

docker-registry.wikimedia.org/releng/quibble-stretch-hhvm

You can find the full list of images by looking though those with quibble in the name from the WMF docker registry. e.g.:

curl -X GET https://docker-registry.wikimedia.org/v2/_catalog | grep quibble

Run quibble with the env file as parameter.

Run:

docker run -it --rm \
  --env-file ./.env \
  -v "$(pwd)"/cache:/cache \
  -v "$(pwd)"/log:/log \
  -v "$(pwd)"/ref:/srv/git:ro \
  -v "$(pwd)"/src:/workspace/src \
  docker-registry.wikimedia.org/releng/quibble-stretch-hhvm:latest

Optionally skip (slow) installation

For repeated runs of the same change, assuming you have once successfully executed the cloning and installation steps, you can omit them by adding –skip-zuul –skip-deps:

docker run -it --rm \
  --env-file ./.env \
  -v "$(pwd)"/cache:/cache \
  -v "$(pwd)"/log:/log \
  -v "$(pwd)"/ref:/srv/git:ro \
  -v "$(pwd)"/src:/workspace/src \
  docker-registry.wikimedia.org/releng/quibble-stretch-hhvm:latest \
  --skip-zuul \
  --skip-deps

Remove LocalSettings.php between runs

Mediawiki ensures that previous installations are not accidentally destroyed by repeated runs of the installer (overwriting configuration values), so it is up to you to always remove the previous run’s configuration file to avoid problems:

rm src/LocalSettings.php

Indices and tables