uftp is a data transfer tool which uses parallel streams to transfer big amounts of data efficiently. It uses client/server architecture so that a user can use the uftp client to transfer data to and from existing uftp servers. DKRZ provides a uftp server (which uses multiple transfer nodes) to enable data transfers to and from DKRZ lustre filesystem.

uftp usage

uftp uses ssh keys to authenticate the user to the uftp server. So to use uftp you have to create a ssh key which is used for data transfer only. To create a key use ssh-keygen, eg:

ssh-keygen -b 4096 -t rsa -f uftpkey

To be able to use this key for authentication to the DKRZ uftp server, you have to upload the key to your personal profile on https://luv.dkrz.de. Choose “User -> public keys” and select the “Add key” button. You simply paste the public key (in our example “uftp-key.pub”) into the text box and select the “uftp” check box, then “register key”.

Keys from luv are transferred to the uftp server every twenty minutes, so after a while you should be able to use uftp.

The uftp client is available in the DKRZ software tree. To use the client, simply load the environment module:

$ module load uftp-client

To get information about the uftp server (and to test if authentication works with your key), use the uftp client:

$ uftp info -u k202066 -i ./uftpkey https://uftp.dkrz.de:9000/rest/auth/HPCDATA
Client identity:    CN=k202066, OU=ssh-local-users
Client auth method: SSHKEY
Auth server type:   AuthServer
  URL base:         https://uftp.dkrz.de:9000/rest/auth/HPCDATA:
  Description:      HPCDATA
  Remote user info: uid=k202066;gid=N/A
  Sharing support:  not available
  Server status:    OK [2 of 2 UFTPD servers available]

To list the content of a directory on the uftp server, use:

$ uftp ls -u k202066 -i ./uftpkey -v https://uftp.dkrz.de:9000/rest/auth/HPCDATA:/mnt/lustre01/scratch/k/k202066
Verbose mode
Using SSH agent
Using SSH key </home/k202066/uftpkey>
drwx 4096 2016-01-28 18:11 test
drwx 4096 2020-01-03 13:09 tmp

 To copy a file from the uftp server, use:

$ uftp cp -u k202066 -i ./uftpkey https://uftp.dkrz.de:9000/rest/auth/HPCDATA:/mnt/lustre01/scratch/k/k202066/myfile.nc ./myfile.nc

 To copy a file to the uftp server, use:

$ uftp cp -u k202066 -i ./uftpkey ./myfile.nc https://uftp.dkrz.de:9000/rest/auth/HPCDATA:/home/dkrz/k202066/myfile.nc

 If your file is big, you can choose to use multiple threads to transfer the data in parallel. This is set with the option “-t <threadnumber>:

$ uftp cp -u k202066 -i ./uftpkey -t4 ./myfile.nc https://uftp.dkrz.de:9000/rest/auth/HPCDATA:/home/dkrz/k202066/myfile.nc

 Other options of uftp can be listed with “uftp –help”. There exist many more options for the “uftp cp” command (eg. to preserve the file modification time, to use multiple tcp streams or even to limit used bandwidth), see “uftp cp -h” for a list of all options.