transferpy package
Subpackages
Submodules
transferpy.Exceptions module
- exception transferpy.Exceptions.ChecksumError[source]
Bases:
ErrorException raised for errors related to Checksums.
- exception transferpy.Exceptions.Error[source]
Bases:
ExceptionBase class for exceptions in this module.
- exception transferpy.Exceptions.FirewallError[source]
Bases:
ErrorException raised for errors related to Firewall.
- exception transferpy.Exceptions.FreeDiskSpaceError[source]
Bases:
ErrorException raised for errors related to (not having enough) free disk space.
- exception transferpy.Exceptions.MySQLError[source]
Bases:
ErrorException raised for errors related to MySQL/MariaDB.
- exception transferpy.Exceptions.NotFoundError[source]
Bases:
ErrorException raised for errors related to missing expected files or resources, such as a host.
- exception transferpy.Exceptions.OverwriteError[source]
Bases:
ErrorException raised for errors related to missing expected files or resources, such as a host.
transferpy.MariaDB module
- class transferpy.MariaDB.MariaDB(remote_execution)[source]
Bases:
objectClass for Transferer MariaDB related command execution
- run_command(host, command)[source]
Executes command on the target host.
- Parameters:
host – command execution target host
command – command to be executed
- Returns:
execution result (returncode, stdout, stderr)
transferpy.Transferer module
Transferer class: main class and entry point for the WMF transfer framework, and what is used internally by the transfer.py command line utility.
For more details, see: https://wikitech.wikimedia.org/wiki/Transfer.py and https://doc.wikimedia.org/transferpy/master/
- class transferpy.Transferer.Transferer(source_host, source_path, target_hosts, target_paths, options={})[source]
Bases:
objectTop class used to implement the tranference of files and databases between hosts, using some fast method- used primarily for cloning data for databases, backup and recoveries, and migrating swift data between hosts, and while it can be used as a general purpose transfer system, it was initially thougt mainly for massive transfer and data generated on the fly (e.g. xtrabackup dumps), and so it uses unix pipes. It is not suitable for transferences that can be paused and continued later.
It can transfer at the moment files, directories (preserving its metadata) and xtrabackup outputs- while compressing and encrypting the data on the wire.
Note: The right spelling of this class should be Tranferrer.
- after_transfer_checks(result, target_host, target_path)[source]
Post-transfer checks: Was the transfer really successful. Yes- return 0; No- return 1 or more.
- attempt_temp_deletion(host, path, file_type)[source]
Delete directory/file if it exists.
- Parameters:
host – host in which path need to be deleted
path – path need to be deleted
file_type – type of the path (‘file’ or ‘dir’)
- calculate_checksum(host, path)[source]
Returns the standard output of the result of calculating the checksum on the target host. If the checksums are different, it throws an exception. If the checkums match, it returns the standard output, if any.
- calculate_checksum_command(host, path)[source]
Returns a list of strings with the command needed to run to check on the given trarget host that the file(s) created are the same as in the source host.
- clean_all_temps(target_host, target_tmp_dir)[source]
Function to clean all the temporary paths if exists in case of any uncaught exception situations.
- Parameters:
target_host – last target host
target_tmp_dir – temporary lock directory at the target
- Returns:
- property compress_command
Returns a string with the command used for compression (gz-compatible) if compression is used and we are not running a decompression command.
- copy_to(target_host, target_path, port)[source]
Copies the source file or dir on the source host to ‘target_host’. ‘target_path’ is assumed to be a directory and the source file or directory will be copied inside.
- create_temp_paths(lock_dir)[source]
Update checksum file paths, and create a temporary directory at the source machine.
- Parameters:
lock_dir – temporary lock directory at the target host
- Returns:
None if successful, else Exception
- property decompress_command
Returns a string with the command used for decompression (gz-compatible) if compression is used.
- property decrypt_command
Returns a command string used to pipe data between the reception and the writing of data to decrypt a stream on the target host, using openssl.
- dir_is_empty(directory, host)[source]
Returns True if the given directory path is really a directory that exists and it is empty.
- disk_usage(host, path, is_xtrabackup=False)[source]
Returns the size used on the filesystem by the file path on the given host, or the aggregated size of all the files inside path and its subdirectories
- property encrypt_command
Returns a command string used to pipe data between the generation and the sending of data to encrypt a stream on the source host, using openssl.
- file_exists(host, path)[source]
Returns true if there is a file or a directory with such path on the remote host given.
- get_datadir_from_socket(socket)[source]
Given a socket name, guess and return the data dir string following WMF conventions.
- handle_transfer_to_one_host(target_host: str, target_path: str, wait_for_source_checksum: bool) int[source]
Perform all per-host operations: firewall open, origin checksum, actual copy, target checksum, after transfer checks, and firewall close :returns: 0 on success, other on error
- has_available_disk_space(host, path, size)[source]
Returns true if the disk space available at host on the given path location is larger than the provided size, otherwise returns false.
- host_exists(host)[source]
Checks the availability of given host by trying to run a noop on it. Returns true if the hosts exists and it is available for remote execution, otherwise returns false.
- Parameters:
host – host to be checked
- Returns:
remote execution run_command result
- property is_decompress
Returns true if the execution involves xtrbackup on the source host, otherwise it returns false.
- is_dir(host, path)[source]
Returns true if the given path is a directory and exists on the given host, otherwise returns false.
- is_socket(host, path)[source]
Returns true if the given path is a socket and exists on the given host, otherwise returns false.
- property is_xtrabackup
Returns true if the copy type is ‘xtrabackup’- that means, it is running a database backup on the source host, otherwise it returns false.
- property mbstream_command
Returns a piped command to add at the end of the receiving stream in the target host to retrieve the files sent from the xtrabackup command on the source host.
- netcat_listen_command(port)[source]
Returns the netcat command used to listen for dataon the target host, given the target port.
- netcat_send_command(target_host, port)[source]
Returns the netcat command used on the source host, including the target host and the target port.
- property parallel_checksum_source_command
command to make source checksum parallel to the file transfer.
- Returns:
string with command to make the checksum
- Type:
Property
- property parallel_checksum_target_command
command to make target checksum parallel to the file transfer.
- Returns:
string with command to make the checksum
- Type:
Property
- property password
Generate and return a random password sufficiently random and secure, in base 64.
- read_checksum(host, path)[source]
Returns a string with the checksum calculated of the given file path on the given host and then deletes the checksum file. It rises an exception if reading or deleting the file errors out.
- run()[source]
Starts the transference of the file (or the directory and all its contents) given on source_path from the source_target machine to all target_hosts hosts, as fast as possible. Returns an array of exit codes, one per target host, indicating if the transfer was successful (0) or not (<> 0).
- run_command(host, command)[source]
It redirects the remote execution of a given command for a given host to the right api executing that.
- run_with_bash(command)[source]
Given a bash command line string, return the full execution list, including the bash execution, to be able to run it on a local host.
- sanity_checks()[source]
Set of preflight checks for the transfer- raise an exception if they are not met.
- property tar_command
Returns the string command used for consolidating a directory into a single file/stream (using tar).
- property untar_command
Returns the string command used for reconstructing a directory back into several files from the incoming stream pipe (using tar).
- property xtrabackup_command
Returns a string with the command used to generate a backup of mysql/mariadb using xtrabackup on the local host- with the appropiate options as reflected on the transferrer properties.
transferpy.transfer module
- class transferpy.transfer.RawOption(prog, indent_increment=2, max_help_position=24, width=None)[source]
Bases:
HelpFormatterClass to format ArgumentParser help
- transferpy.transfer.assign_default_options(options)[source]
Assign default values if the given options dictionary is missing some required arguments.
- Parameters:
options – given options
- Returns:
- transferpy.transfer.option_parse()[source]
Parses the input parameters and returns them as a list.
- Returns:
sender host, sender path, receiver hosts, receiver paths, other options
- transferpy.transfer.parse_configurations(config_file)[source]
Parses the configuration file parameters.
- Returns:
parser object
- transferpy.transfer.setup_logger(verbose)[source]
Setup a logger named transferpy. The logger level is set based on the verbose value. If verbose is true, set the logger to DEBUG level, else INFO level. This logger is globally available in the package and can be accessed using logging.getLogger(‘transferpy’).
- Parameters:
verbose – verbose boolean variable
- Returns: