lftp is a file transfer program designed to transfer bigger amount of data (big files).
lftp is used on top of file transfer protocols such as ftp, http, sftp, fish, torrent. It allows to parallelize the data streams.
See also the lftp webpage (https://lftp.yar.ru/).
Use lftp carefully, don't overload the connections, respect the bandwidth needs of other users! Set your parameters carefully - especially the number of threads (-P <threads>).
If you want to use lftp from your workstation you have to install it. If your workstation acts as server in the data transfer, no installation is necessary. It is recommended to use at minimum LFTP version 4.5.3.
Most distributions have an lftp version installed. However it is recommended to install a recent version from: https://lftp.yar.ru/get.html
MacPorts (https://www.macports.org/) contains a recent version of lftp.
NOT TESTED! There is a Windows version available under https://nwgat.ninja/lftp-for-windows/ . Since openssh is officially available for Windows 10 it is strongly recommended to use Windows 10 only.
Usage on ID-HPC clusters
For smaller and biggerfiles
To copy (download) the content of a folder from a remote system to a local path inside Leonhard with 8 parallel file sessions:
lftp sftp://firstname.lastname@example.org -e "mirror -P 8 --use-pget-n=1 /data/smallfiles /cluster/scratch/username/test1/; exit”
While -P <N>: number of threats; --use-pget-n=<n>: number of chunks per file (default 1)
To copy (download) the content of a folder from a remote system to a local path in Leonhard with 12 parallel file sessions:
lftp sftp://email@example.com -e "mirror -P 12 --use-pget-n=1 /data/smallfiles /cluster/scratch/username/test2/; exit”
For bigger files
Bigger files minimum > 20GB (test cases), real use cases for > 100GB.
LFTP copy (download) 4 files in parallel and in addition split each file into 4 chunks during transfer (16 flows in total):
lftp sftp://firstname.lastname@example.org -e "mirror -P 4 --use-pget-n=4 /data/bigfiles /cluster/scratch/username/test3/; exit”
LFTP copy (download) 8 files in parallel and in addition split each file into 4 chunks during transfer (32 flows in total):
lftp sftp://email@example.com -e "mirror -P 8 --use-pget-n=4 /data/bigfiles /cluster/scratch/username/test4/; exit"
lftp has built-in mirror which can download or update a whole directory tree. There is also reverse mirror (mirror -R) which uploads or updates a directory tree on server. Mirror can also synchronize directories between two remote servers, using FXP if available. (copied from man page)
Manpages are your friend. man lftp serves as reference manual. The web version of the man page is on the lftp website