Data Transfer#

Depending on the specific data transfer requirements users can rely on a variety of methods to move data from/to DKRZ systems as well as between DKRZ systems. Below an incomplete overview of data transfer tools and protocols is given. For some tools links to the pages that introduce them in more details are provided.

  • uftp is a data transfer tool from the Unicore environment. It offers efficient data transfers using parallel data streams and multiple threads. The uftp client runs on different operating systems and can be used with ssh keys for secure authentication. This is the preferred tool for large data transfers.

  • rsync is widely used for synchronisation of remote or local directories (e.g. backups, mirroring) and as an enhanced copy command. It employs delta-transfer algorithm to send only differences between source files and existing destination files.

  • scp, sftp are basic transfer tools that use SSH protocol to copy data between remote machines.

  • slk is user interface to the StrongLink software and allows the user to interact with the Stronglink HSM (tape archive)

  • swift is a command line tool to copy data to/from Swift Object Storage at DKRZ. Particularly suitable for upload and download of large data sets.

  • wget, curl are tools for client-side URL transfers. Mainly used for recursive download of data from servers over HTTP or HTTPS.