Usage
transfer.py is installed (via Puppet) on PATH on WMF production infrastructure on the Cumin hosts, and has to run as root (like cumin).
transfer.py –help
transfer is a Python 3 framework intended to move large files or directory trees between WMF production hosts in an efficient way.
usage: transfer.py [-h] [--config CONFIG_FILE] [--port PORT]
[--type {file,xtrabackup,decompress}]
[--compress | --no-compress] [--encrypt | --no-encrypt]
[--checksum | --no-checksum]
[--parallel-checksum | --no-parallel-checksum]
[--stop-slave] [--verbose]
source target [target ...]
Positional Arguments
- source
Fully qualified domain of the host where the files to be copied are currently located, the symbol ‘:’, and a file or directory path of such files (e.g. sourcehost.wm.org:/srv ). There can be only one source host and path.
- target
Fully qualified domain of the hosts (separated by spaces) where the files to be copied, each one with its destination absolute path directory, separated by ‘:’.There must be at least one target. If more than one target is defined, it will be copied to all of them. The target path MUST be a directory.
Named Arguments
- --config
Configuration file path. If found, it will set as the execution defaults unless overridden on the command line. Default: /etc/transferpy/transferpy.conf
- --port
Port used for netcat listening on the receiver machine. By default, transfer selects a free port available in the receiver machine from the range 4400 to 4500
- --type
Possible choices: file, xtrabackup, decompress
raw|file: regular file or directory recursive copy (Default) xtrabackup: runs mariabackup on source decompress: a tarball is transmitted as is and decompressed on target
- --compress
Use pigz to compress stream using gzip format (ignored on decompress mode) (Default)
- --no-compress
Do not use compression on streaming
- --encrypt
Enable compression - send data using openssl and algorithm chacha20 (Default)
- --no-encrypt
Disable compression - send data using an unencrypted stream
- --checksum
Generate a checksum of files before transmission which will be used for checking integrity after transfer finishes. (This only works for file transfers) (Default)
- --no-checksum
Disable checksums
- --parallel-checksum
Generate checksum parallel to the transmission for data integrity check (ignored if –checksum is enabled). –parallel_checksum is faster than –checksum but less reliable
- --no-parallel-checksum
Disable parallel checksum (Default)
- --stop-slave
Only relevant if on xtrabackup mode: attempt to stop slave on the mysql instance before running xtrabackup, and start slave after it completes to try to speed up backup by preventing many changes queued on the xtrabackup_log. By default, it doesn’t try to stop replication.
- --verbose
Outputs relevant information about transfer + information about Cuminexecution. By default, the output contains only relevant information about the transfer.
Thank you! Full documentation at: https://wikitech.wikimedia.org/wiki/Transfer.py