Usage

transfer.py is installed (via Puppet) on PATH on WMF production infrastructure on the Cumin hosts, and has to run as root (like cumin).

transfer.py –help

transfer is a Python 3 framework intended to move large files or directory trees between WMF production hosts in an efficient way.

usage: transfer.py [-h] [--config CONFIG_FILE] [--port PORT]
                   [--type {file,xtrabackup,decompress}]
                   [--compress | --no-compress] [--encrypt | --no-encrypt]
                   [--checksum | --no-checksum]
                   [--parallel-checksum | --no-parallel-checksum]
                   [--stop-slave] [--verbose]
                   source target [target ...]

Positional Arguments

source

Fully qualified domain of the host where the files to be copied are currently located, the symbol ‘:’, and a file or directory path of such files (e.g. sourcehost.wm.org:/srv ). There can be only one source host and path.

target

Fully qualified domain of the hosts (separated by spaces) where the files to be copied, each one with its destination absolute path directory, separated by ‘:’.There must be at least one target. If more than one target is defined, it will be copied to all of them. The target path MUST be a directory.

Named Arguments

--config

Configuration file path. If found, it will set as the execution defaults unless overridden on the command line. Default: /etc/transferpy/transferpy.conf

--port

Port used for netcat listening on the receiver machine. By default, transfer selects a free port available in the receiver machine from the range 4400 to 4500

--type

Possible choices: file, xtrabackup, decompress

raw|file: regular file or directory recursive copy (Default) xtrabackup: runs mariabackup on source decompress: a tarball is transmitted as is and decompressed on target

--compress

Use pigz to compress stream using gzip format (ignored on decompress mode) (Default)

--no-compress

Do not use compression on streaming

--encrypt

Enable compression - send data using openssl and algorithm chacha20 (Default)

--no-encrypt

Disable compression - send data using an unencrypted stream

--checksum

Generate a checksum of files before transmission which will be used for checking integrity after transfer finishes. (This only works for file transfers) (Default)

--no-checksum

Disable checksums

--parallel-checksum

Generate checksum parallel to the transmission for data integrity check (ignored if –checksum is enabled). –parallel_checksum is faster than –checksum but less reliable

--no-parallel-checksum

Disable parallel checksum (Default)

--stop-slave

Only relevant if on xtrabackup mode: attempt to stop slave on the mysql instance before running xtrabackup, and start slave after it completes to try to speed up backup by preventing many changes queued on the xtrabackup_log. By default, it doesn’t try to stop replication.

--verbose

Outputs relevant information about transfer + information about Cuminexecution. By default, the output contains only relevant information about the transfer.

Thank you! Full documentation at: https://wikitech.wikimedia.org/wiki/Transfer.py