transferpy is a Python 3 framework and command-line utility intended to efficiently move large files or directory trees between WMF production hosts, as well as to provide a low-level tool to backup and recover MySQL/MariaDB servers.


Before there was a backup and provisioning workflow, new database hosts (or hosts that needed rebuild, e.g. after a crash) were setup in different, completely manual ways. Mostly using netcat.

Before cumin was available, work started on a way to automate that (Task T156462) with the aim to add a programmable api, consistency, speedup, handling the firewall, and generally stop doing things manually. Used at first to perform cold database copies.

Once cumin was deployed, and logical backups were in place, was extended to be used for hot database backups and recoveries, as a building block of the recovery system.

Technical details

The original spec for a transfer script was using multicast or bittorrent protocols for fast recovery of multiple hosts. For the time, however, basically cumin is run so that netcat is run on source, and another instance listening on target host, and either a single file or a tar copy of a directory is piped through. Optionally, compression with pigz can be used, as well as encryption with openssl. pigz can improve enormously the speed of the transfer, as database can be compressed as much as 5 times, reducing the total bandwidth used. Data can also be checksummed, but that adds some overhead at the beginning of the transfer.

At the Wikimedia Foundation infrastructure, cumin is being used as the remote execution framework, but others are also available and can be made to work. However, for things like mysql transfers, certain things like mysql port assignation and paths are assumed to be in a specific places.

For MySQL, two modes (in addition to the default “file”) were added:

  • xtrabackup: where the source of the backup is not obtained from the filesystem, but from a mariabackup run with streaming (xbstream) from a running MariaDB server. In this mode, the mysql server socket is used as the origin path. It can optionally stop replication in cases where that may speed up the copy/preparation process. It only transfer the files, it does not prepare or touch in any way the generated files (that is considered out of scope of this framework, as one may want to wait to do that for incremental backups or other reasons).

  • decompress: where the source is a precompressed tar.gz file containing a single directory, intended with the same format as the snapshot tarballs generated by the backup system (prepared xtrabackup datadirs), and result in a decompressed directory (very similar to “file” operation, but without having to compress on source)

In addition to the automation of all commands (iptables, tar, openssl, pigz, nc, remote execution), does important sanity checks, like making sure it does not overwrite existing files (it aborts before transfer) and making sure it has enough disk space on the selected directory.