GSoC 2020
Student: Ajumal P A
Separate WMFMariadbpy and transferpy to its own repositories.
Modularize transferpy into three modules and a RemoteExecution:
The
Transferer
class is responsible for the act on the user arguments and make the send/receive possible.The
Firewall
class is for open/close the ports in the iptables to receive the data by the receiver machines.The RemoteExecution (now moved to the wmfmariadbpy repo) is the module responsible for the execution of command on the remote machines.
transfer framework mainly uses the Cumin execution.
Give proper comments to newly created modules.
Make –help option useful to the user.
Make documentation using Sphinx (This).
Remove unimportant output of Cumin from the console
Improve the transferpy logs using the python logging module and also by adding more error messages.
Use a new parameter verbose in order to understand the user logging level need.
Detect a free port automatically on the target machine using netstat for the netcat to listen
Firewall.find_available_port
.Resolve concurrency issues related to the port finding using a lock based approach.
Make checksum parallel to the actual transfer if the user enables newly added parallel-checksum option. (This checksum only detects network integrity issues)
Make the normal checksum parallel to actual transfer at source machine using multiprocessing. (This checksum detects both network and disk related integrity issues)
Resolve concurrency issue of checksum related temporary file names.
Benchmark and prove the efficiency of checksum options.
Enable configuration file option for user arguments.
Since temporary files are being created by transferpy for locking and checksumming, implemented a way to clean them properly at the end of execution even if any errors/exception happened in the middle of transfer.
Make Debian transferpy package.
Add proper tests and documentation for the new features.
Release transferpy-1.0 deb package with all the above features.