Data Transfer#

file version: 9 Jul 2026

Overview#

The interface to the HSM system is called ATLAS. It provides both a command-line interface atlas and a web interface. The Versity system offers additional interfaces for data exchange – i.e. mount points on Levante and an S3 Gateway – which will be made available at a later point in time. For the time being, ATLAS remains the only user interface.

ATLAS is meant as the central tool for data transfer between tape archive and Lustre filesystem. Using ATLAS requires a login with your default DKRZ credentials.

If you are looking for a quick guide for the migration from slk and/or slk_helpers to new commands of the new software, please have a look here.

Transfer: When to use which tool?#

Levante/Lustre <-> Tape: ATLAS#

The default tool for transferring data between the Lustre filesystem and the DKRZ tape archive is ATLAS. The cli atlas and the ATLAS WebGUI offer nearly the same set of features. atlas can be used interactively but also in scripts. You need to login with your default DKRZ user credentials.

S3 <-> Tape: Versity S3 Gateway (LATER)#

When you want to transfer data between the DKRZ S3 system and the DKRZ tape archive, we recommend using the Versity S3 endpoint with a tool that allows S3-to-S3 transfers like rclone. You need to generate an S3 key and secret via LUV.

Look into one file from tape: Mount Point (LATER)#

When you want to peek into a single file or process it, you can use the read only mounts /arch, /double and /doku to access it. Please stage (“recall” in StrongLink) the file in advance via atlas stage <file_path>.

Outside world <-> Tape: depends#

When you want to transfer data between the outside world and DKRZ tape archive, please use the DKRZ S3 storage as an intermediate step. If you need to transfer multiple TB between DKRZ and another side once, we are able to set up an S3 endpoint which is open to the world. For the reasons stated above, this will only be done in a very few cases for the time being. Additionally, we will elaborate on the usage of Globus Transfer incombination with the Versity system once we migrated successfully and resolved major teething problems.

File transfer via ATLAS#

ATLAS offers three modes for file transfer:

  • cp (recommended; “archival” or “retrieval”):

    • transfer a file within the HSM system or between HSM system and the Lustre filesystem (/scratch: both directions; /work: read-only)

    • ATLAS handles the transfer for you

    • atlas/WebGUI can be exited after transfer was started

    • check jobs in ATLAS to get the status of running file transfers

    • the cp command requires absolute paths

    • automatic checksum verification

  • upload (”archival”):

    • transfer a file which is available on your “local” filesystem to the HSM system

    • atlas/WebGUI has to run while the transfer is ongoing

  • download (”retrieval”):

    • transfer a file from the HSM system to your “local” filesystem

    • atlas/WebGUI has to run while the transfer is ongoing

The transfer via atlas cp / WebGUI’s copy is the recommended transfer mode. It is much more efficient than upload and download and should only be used for very specific use cases.

Warning

atlas mv performs a real move between filesystems. After the transfer completes successfully, it attempts to remove the source data.

Be careful when using atlas mv to archive data to HSM.

  • atlas mv /work/b123456/mydata.nc /hsm/archive/b123456/mydata.nc copies the file to HSM, but the original file remains in /work because the filesystem is mounted read-only and cannot be modified.

  • atlas mv /scratch/b123456/mydata.nc /hsm/archive/b123456/mydata.nc copies the file to HSM and removes the original from /scratch, because /scratch is writable.

If you want to keep the source data, use atlas cp instead of atlas mv.

Examples#

Copy a folder recursively from /work into the archive.

$ atlas cp --recursive /work/ka1209/k204221/output_42 /hsm/arch/ka1209/k204221/

Create a destination folder and copy two files from /scratch into the archive.

$ atlas mkdir /hsm/arch/ka1209/single_files
$ atlas cp /scratch/k/k204221/file_01.txt /work/k/k204221/file_02.txt /hsm/arch/ka1209/single_files/

Retrieve files from the archive to /scratch. For the time being, atlas stage is to be run before atlas cp can be used for retrievals.

$ atlas stage --recursive /hsm/arch/ka1209/k204221/output_42
$ atlas cp --recursive /hsm/arch/ka1209/k204221/output_42 /scratch/k/k204221

Cache, Online, Offline, Stage, Release!?#

Storage of data on tape is cheap compared to storage on HDD, SSD or similar. Therefore, the DKRZ data archive uses tapes for data storage. However, accessing data from tape is relatively slow compared to other storage types. To overcome this disadvantage, a hierarchical storage management (HSM) system is used at DKRZ which temporarily stores additional copies of archived files on faster storage systems than tape. The new Versity system may store copies of files in three categories of storages. Each storage category can be associated with a time span until requested data are available.

  • primary SSD cache: file instantly available

  • intermediate HDD storage: file accessed delayed by a few seconds

  • tape: file access delayed by a few minutes because file needs to be read from tape

Copies of one file might be stored in multiple storages and storage categories. The file state online, warm or offline indicates where the fastest available copy of the respective file is located:

  • online: 1x file copy in primary SSD cache (possibly also on tape)

  • warm: no file copy in primary SSD cache; 1x file copy in intermediate storage

  • offline: file copies only on tape

The primary cache and the intermediate storage are cleaned up automatically. You can manually tell Versity to remove a file from the cache by releasing it. Commonly, you do not need to do this because the cache is cleaned up automatically quite well.

If you archive a file, it is written into the cache and, later, from there to tape. “Later” in this context can be minutes or hours depending on the load on the system and the availability of tape drives. This process cannot be controlled by you. A file cannot be removed from the cache if it has not been written to tape.

You can stage files as follows:

$ atlas stage <FILE>

# OR

$ atlas stage --recursive <DIRECTORY>

The cleanup of the cache starts when the fill state reaches an upper threshold. Then, the oldest files are removed first – based on the last access time and the last staging time – until the fill state reaches a lower threshold. Thus, the atlas stage command also acts like touch. There is no guarantee that a file remains cached for X hours after running atlas stage. Please do not run the command permanently for all of your files in a loop. If all users were doing this … .

File access via HSM mount points on Levante#

It is planned to make /hsm/arch, /hsm/double and /hsm/doku available via equally named mount points on Levante. The optimal configuration for this setup still has to be clarified and, hence, these mount points are not available from the beginning. We will notify you when they are set up.

File transfer via S3#

If you need to transfer data between the Lustre filesystem and the tape archive, please use ATLAS. However, if data should be transferred between an S3 bucket and the tape archive, the Versity S3 Gateway is best suited.

The archival of data can be done with a S3 client of your choice. Retrieval of files requires a S3 tool which supports the Amazon Glacier command set. Details will be published later on.

File transfer from/to outside DKRZ#

Please contact us via support@dkrz.de

On this page