Data Transfer

Depending on the specific data transfer requirements users can rely on a variety of methods to move data from/to DKRZ systems as well as between DKRZ systems. Below an incomplete overview of data transfer tools and protocols is given. For some tools links to the pages that introduce them in more details are provided.

  • uftp is a data transfer tool from the Unicore environment. It offers efficient data transfers using parallel data streams and multiple threads. The uftp client runs on different operating systems and can be used with ssh keys for secure authentication.

  • GridFTP is a powerful, secure and reliable data transfer protocol optimized for transfer of large data sets. The most popular implementation is being developed as part of the Globus Toolkit, a bundle of tools for grid computing.

  • rsync is widely used for synchronisation of remote or local directories (e.g. backups, mirroring) and as an enhanced copy command. It employs delta-transfer algorithm to send only differences between source files and existing destination files.

  • scp, sftp are basic transfer tools that use SSH protocol to copy data between remote machines.

  • lftp is command line client for file transfer that supports several network protocols (e.g. FTP(S), HTTP(S), SFTP). LFTP provides mirroring capabilities, which can be used to download, upload or update a complete directory tree.

  • slk is user interface to the StrongLink software and allows the user to interact with the Stronglink HSM (tape archive)

  • swift is a command line tool to copy data to/from Swift Object Storage at DKRZ. Particularly suitable for upload and download of large data sets.

  • wget, curl are tools for client-side URL transfers. Mainly used for recursive download of data from servers over HTTP or HTTPS.