Quibble: a test runner for MediaWiki¶

Quibble will clone the specific repository being tested, MediaWiki, and any dependencies. Then all available tests are run, beginning with basic lint checks and culminating in browser tests for each MediaWiki extension. Specific tests can be included or excluded as needed.

Everything is performed by a single command, quibble.

Running quibble requires python 3 and the following tools, or you can run in the provided docker image.

Chromium
composer
NodeJS
npm
php
Xvfb

Quick Start¶

Full build and run, with no caching:

docker build --tag quibble .
docker run -it --rm quibble

Which runs tests with php7.0, MariaDB and using mediawiki/vendor.git to provide PHP libraries.

You could instead run tests with dependencies provided by composer install and use SQLite as a database backend:

docker run -it quibble  --packages-source composer --db sqlite

Wikimedia maintains Docker containers intended to be used for its continuous integration system, for example:

docker pull docker-registry.wikimedia.org/releng/quibble-buster-php74:latest

The source is on Gerrit https://gerrit.wikimedia.org/g/integration/config under the dockerfiles directory, where you’ll also find other images with slight variations such as other PHP versions.

Further documentation can be found on https://doc.wikimedia.org/quibble/ .

Setup¶

Docker container¶

Get the latest image being run by Wikimedia CI:

docker pull docker-registry.wikimedia.org/releng/quibble-buster-php74:latest

Quibble clones the repositories from Gerrit, and may load additional dependencies using composer and npm. At the end of the run, the container will be removed as well as all of the downloaded content. To make it faster, you should provide local copies of the git repositories as a volume attached to the container.

To avoid cloning MediaWiki over the network, you should initialize local bare git repositories to be used as a reference for git to copy them from:

mkdir -p ref/mediawiki/skins
git clone --bare https://gerrit.wikimedia.org/r/mediawiki/core ref/mediawiki/core.git
git clone --bare https://gerrit.wikimedia.org/r/mediawiki/vendor ref/mediawiki/vendor.git
git clone --bare https://gerrit.wikimedia.org/r/mediawiki/skins/Vector ref/mediawiki/skins/Vector.git

The Docker containers have XDG_CACHE_HOME=/cache set which is recognized by package managers. Create a cache directory writable by any user:

mkdir cache
chmod 777 cache

Commands write logs into /workspace/log, you can create one on the host and mount it in the container:

mkdir -p log
chmod 777 log

You might also want to reuse the installed sources between runs. The container has the source repository under /workspace/src:

mkdir -p src
chmod 777 src

The directory tree on the host will looks like:

.
├── cache/
├── log/
├── src/
└── ref/
    └── mediawiki/
        ├── core.git/
        ├── skins/
        │   └── Vector.git/
        └── vendor.git/

When running the Docker container, mount the directories from the host:

Host dir	Container dir	Docker run argument
`./cache/`	`/cache`	`-v "$(pwd)"/cache:/cache`
`./log/`	`/workspace/log`	`-v "$(pwd)"/log:/workspace/log`
`./ref/`	`/srv/git`	`-v "$(pwd)"/ref:/srv/git:ro`
`./src/`	`/workspace/src`	`-v "$(pwd)"/src:/workspace/src`

The final command:

docker run -it --rm \
  -v "$(pwd)"/cache:/cache \
  -v "$(pwd)"/log:/workspace/log \
  -v "$(pwd)"/ref:/srv/git:ro \
  -v "$(pwd)"/src:/workspace/src \
  docker-registry.wikimedia.org/releng/quibble-buster-php74:latest

Quibble will then do the initial cloning of repositories reusing bare repositories from ref, being local it is arguably faster than transferring everything from Gerrit. The composer install and npm install will save the downloaded packages to cache which speed up the next run.

Finally, having /src mounted from the host, lets one reuse the installed wiki. One can later skip cloning/checking out the repositories by passing --skip-zuul and skip installing composer and npm dependencies with --skip-deps. For other options see: Quibble CLI usage.

TESTING¶

Coverage report:

tox -e cover && open cover/index.html

quibble.yaml¶

Since version 1.5.0, Quibble will look for a quibble.yaml file in the root of the project it is testing.

The current supported configuration options are:

# "early warning" related functionality, when Quibble fails a job
# (e.g. 'composer-test' or 'npm-test' exit with a non-zero code)
# Quibble will read this configuration to send to an external
# HTTP endpoint. See also the --reporting-url option.
earlywarning:
    # Quibble passes both the "should_vote" and "should_comment"
    # values to an external HTTP endpoint. An application at
    # that endpoint can then potentially make a comment in
    # a code review system with a verification vote and/or
    # a comment with the status of the failed job.
    should_vote: 1
    should_comment: 1

Reproducing a CI build¶

Quibble allows you to locally replicate test runs as you witnessed them in CI. This can be useful when debugging integration with projects you are not normally involved with and don’t keep a local copy of, or other obscure problems.

Create a .env file to specify the variables needed to replicate a CI run.¶

You can find them in the Jenkins job parameters, [for example](https://phab.wmfusercontent.org/file/data/intwp5iddudl53ec24uu/PHID-FILE-4h2a5udx4sjoodnitahl/jenkins_params.png):

BASE_LOG_PATH=67/545967/1
EXT_DEPENDENCIES=mediawiki/extensions/AbuseFilter\nmediawiki/extensions/AntiSpoof\nmediawiki/extensions/Babel\nmediawiki/extensions/CheckUser\nmediawiki/extensions/CirrusSearch\nmediawiki/extensions/Cite\nmediawiki/extensions/CiteThisPage\nmediawiki/extensions/CodeEditor\nmediawiki/extensions/ConfirmEdit\nmediawiki/extensions/ContentTranslation\nmediawiki/extensions/Echo\nmediawiki/extensions/Elastica\nmediawiki/extensions/EventLogging\nmediawiki/extensions/FileImporter\nmediawiki/extensions/Flow\nmediawiki/extensions/Gadgets\nmediawiki/extensions/GeoData\nmediawiki/extensions/GlobalCssJs\nmediawiki/extensions/GlobalPreferences\nmediawiki/extensions/GuidedTour\nmediawiki/extensions/ImageMap\nmediawiki/extensions/InputBox\nmediawiki/extensions/Interwiki\nmediawiki/extensions/JsonConfig\nmediawiki/extensions/MobileApp\nmediawiki/extensions/MobileFrontend\nmediawiki/extensions/NavigationTiming\nmediawiki/extensions/ParserFunctions\nmediawiki/extensions/PdfHandler\nmediawiki/extensions/Poem\nmediawiki/extensions/SandboxLink\nmediawiki/extensions/SiteMatrix\nmediawiki/extensions/SpamBlacklist\nmediawiki/extensions/TemplateData\nmediawiki/extensions/Thanks\nmediawiki/extensions/TimedMediaHandler\nmediawiki/extensions/Translate\nmediawiki/extensions/UniversalLanguageSelector\nmediawiki/extensions/VisualEditor\nmediawiki/extensions/WikiEditor\nmediawiki/extensions/Wikibase\nmediawiki/extensions/WikibaseCirrusSearch\nmediawiki/extensions/WikibaseMediaInfo\nmediawiki/extensions/cldr
EXT_NAME=MobileFrontend
LOG_PATH=67/545967/1/test/wmf-quibble-vendor-mysql-php74-docker/55820fc
MW_COMPOSER_MERGE_MW_IN_VENDOR=1
SKIN_DEPENDENCIES=mediawiki/skins/MinervaNeue\nmediawiki/skins/Vector
ZUUL_CHANGE=545967
ZUUL_CHANGE_IDS=545967,1
ZUUL_CHANGES=mediawiki/extensions/MobileFrontend:master:refs/changes/67/545967/1
ZUUL_COMMIT=c28d0377ea56884905e57cf81ca422cac07ece98
ZUUL_PATCHSET=1
ZUUL_PIPELINE=test
ZUUL_PROJECT=mediawiki/extensions/MobileFrontend
ZUUL_REF=refs/zuul/master/Zbe0ebb207a1d4c33935e2a0cf8db9a1c
ZUUL_URL=git://contint2001.wikimedia.org
ZUUL_UUID=55820fcebdae4c4291e95f01d4b3f987
ZUUL_VOTING=1

Not all of the variables visible in the Jenkins jobs parameters are needed. The important ones are:

EXT_DEPENDENCIES
SKIN_DEPENDENCIES
ZUUL_PROJECT
ZUUL_REF
ZUUL_BRANCH

EXT_DEPENDENCIES and SKIN_DEPENDENCIES are deprecated and will emit a warning. Instead the list of repositories should be passed to Quibble as arguments:

quibble mediawiki/extensions/BetaFeatures mediawiki/skins/MinervaNeue [...]

At the moment only builds with one change set can be reproduced locally because git://contint2001.wikimedia.org is not accessible remotely.

Choose the right Docker image.¶

You must also choose the correct quibble image for the base OS and php interpreter to mirror the job: e.g. Debian Buster and php 7.4:

docker-registry.wikimedia.org/releng/quibble-buster-php74

You can find the full list of images by looking though those with quibble in the name from the WMF docker registry. e.g.:

curl -X GET https://docker-registry.wikimedia.org/v2/_catalog | grep quibble

Run quibble with the env file as parameter.¶

Run:

docker run -it --rm \
  --env-file ./.env \
  -v "$(pwd)"/cache:/cache \
  -v "$(pwd)"/log:/log \
  -v "$(pwd)"/ref:/srv/git:ro \
  -v "$(pwd)"/src:/workspace/src \
  docker-registry.wikimedia.org/releng/quibble-buster-php74:latest

Optionally skip (slow) installation¶

For repeated runs of the same change, assuming you have once successfully executed the cloning and installation steps, you can omit them by adding –skip-zuul –skip-deps:

docker run -it --rm \
  --env-file ./.env \
  -v "$(pwd)"/cache:/cache \
  -v "$(pwd)"/log:/log \
  -v "$(pwd)"/ref:/srv/git:ro \
  -v "$(pwd)"/src:/workspace/src \
  docker-registry.wikimedia.org/releng/quibble-buster-php74:latest \
  --skip-zuul \
  --skip-deps

Remove LocalSettings.php between runs¶

Mediawiki ensures that previous installations are not accidentally destroyed by repeated runs of the installer (overwriting configuration values), so it is up to you to always remove the previous run’s configuration file to avoid problems:

rm src/LocalSettings.php