TODO: symbolic links not transfered

Data Transfer#

file version: 9 Jul 2026

Overview#

The interface to the HSM system is called ATLAS. It provides both a command-line interface atlas and a web interface. The Versity system offers additional interfaces for data exchange – i.e. mount points on Levante and an S3 Gateway – which will be made available at a later point in time. For the time being, ATLAS remains the only user interface.

ATLAS is meant as the central tool for data transfer between tape archive and Lustre filesystem. Using ATLAS requires a login with your default DKRZ credentials.

If you are looking for a quick guide for the migration from slk and/or slk_helpers to new commands of the new software, please have a look here.

Transfer: When to use which tool?#

Levante/Lustre <-> Tape: ATLAS#

The default tool for transferring data between the Lustre filesystem and the DKRZ tape archive is ATLAS. The cli atlas and the ATLAS WebGUI offer nearly the same set of features. atlas can be used interactively but also in scripts. You need to login with your default DKRZ user credentials.

S3 <-> Tape: Versity S3 Gateway (LATER)#

When you want to transfer data between the DKRZ S3 system and the DKRZ tape archive, we recommend using the Versity S3 endpoint with a tool that allows S3-to-S3 transfers like rclone. You need to generate an S3 key and secret via LUV.

Look into one file from tape: Mount Point (LATER)#

When you want to peek into a single file or process it, you can use the read only mounts /arch, /double and /doku to access it. Please stage (“recall” in StrongLink) the file in advance via atlas stage <file_path>.

Outside world <-> Tape: depends#

When you want to transfer data between the outside world and DKRZ tape archive, please use the DKRZ S3 storage as an intermediate step. If you need to transfer multiple TB between DKRZ and another side once, we are able to set up an S3 endpoint which is open to the world. For the reasons stated above, this will only be done in a very few cases for the time being. Additionally, we will elaborate on the usage of Globus Transfer incombination with the Versity system once we migrated successfully and resolved major teething problems.

File transfer via ATLAS#

ATLAS offers four modes for file transfer:

  • cp (recommended; “archival” or “retrieval”):

    • transfer a file within the HSM system or between HSM system and the Lustre filesystem (/scratch: both directions; /work: read-only)

    • ATLAS handles the transfer for you

    • atlas/WebGUI can be exited after transfer was started

    • check jobs in ATLAS to get the status of running file transfers

    • the cp command requires absolute paths

    • automatic checksum verification

    • symbolic links are ignored without warning or error

  • upload (”archival”):

    • transfer a file which is available on your “local” filesystem to the HSM system

    • atlas/WebGUI has to run while the transfer is ongoing

  • download (”retrieval”):

    • transfer a file from the HSM system to your “local” filesystem

    • atlas/WebGUI has to run while the transfer is ongoing

  • move (”archival” or “retrieval”):

    • like cp / copy but removes source file after successful transfer

The transfer via atlas cp / WebGUI’s copy is the recommended transfer mode. It is much more efficient than upload and download which should only be used for very specific use cases.

Warning

cp ignores symbolic links without throwing a warning or an error. This is filed as a bug and we are expecting a fix with the upcoming major release of atlas (end of Aug. 2026).

Warning

A mv within /hsm is a pure metadata operation. In all other cases, atlas mv performs a real move between filesystems. After the transfer completes successfully, it attempts to remove the source data. If you accidentally moved data from /hsm to /work or /scratch and want it back in /hsm, please to not copy it back but notify support@dkrz.de . We’ll try to restore the original tape copy.

Be careful when using atlas mv to archive data to HSM.

  • atlas mv /work/b123456/mydata.nc /hsm/archive/b123456/mydata.nc copies the file to HSM, but the original file remains in /work because the filesystem is mounted read-only and cannot be modified.

  • atlas mv /scratch/b123456/mydata.nc /hsm/archive/b123456/mydata.nc copies the file to HSM and removes the original from /scratch, because /scratch is writable.

If you want to keep the source data, use atlas cp instead of atlas mv.

When you want to transfer data from /hsm to somewhere else, all four transfer commands can only transfer cached files (online) and do not automatically stage files from tape. This will change in ATLAS version 4 which is to be released in end of August 2026. Until then, please run atlas stage manually to get the files back into the cache. The command will submit a stage job which status can be checked via atlas jobs get. You can run the transfer command of your choice while the staging is running. However, only those file, which were cached when the transfer command was started, will be transfered. Therefore, the transfer command has to be run at least once after the stage job ended. We provide a script template further below where you just need to update src and dst. This is only required until the release of ATLAS version 4 in the end of August.

Examples#

Copy a folder recursively from /work into the archive.

# submit copy job
$ atlas cp --recursive /work/ka1209/k204221/output_42 /hsm/arch/ka1209/k204221/
copy job submitted: 175
# check job status
$ atlas jobs get 175
Job ID:      175
Type:        copy
State:       running
User:        k204221 (uid=25301)
Node:        vsy-srv-05
Source res.: work
Dest res.:   scratch
Items:       0 / 2
Created:     2026-07-24 13:48:59
Started:     2026-07-24 16:14:12

Create a destination folder and copy two files from /scratch into the archive.

$ atlas mkdir /hsm/arch/ka1209/single_files
$ atlas cp /scratch/k/k204221/file_01.txt /work/k/k204221/file_02.txt /hsm/arch/ka1209/single_files/
copy job submitted: 176
# ...

Retrieve files from the archive to /scratch. For the time being, atlas stage is to be run before atlas cp can be used for retrievals. The cp in the next ATLAS release expected for mid to end of August 2026 will do staging automatically.

# start stage job
$ atlas stage --recursive /hsm/arch/ka1209/k204221/output_42
staging job submitted: 177
# check job status
$ atlas jobs get 16268
Job ID:      16268
Type:        stage
State:       completed
User:        k204221 (uid=25301)
Node:        vsy-srv-05
Resource:    hsm
Items:       1 / 1
Created:     2026-07-23 08:32:17
Started:     2026-07-23 21:24:00
Completed:   2026-07-24 01:44:45
# start copy job when files have been staged
$ atlas cp --recursive /hsm/arch/ka1209/k204221/output_42 /scratch/k/k204221
copy job submitted: 178
# check copy job status ...

script template retrieval: stage + copy#

Template for file retrieval combining atlas stage and atlas cp. The copy process is started already while the staging is still ongoing. As soon as ATLAS version 4 will be release in the end of August 2026, atlas cp will automatically stage files and this template becomes unnecessary.

# TODO: adapt these two lines!
src=/hsm/arch/bm0146/k204221/my_data
dst=/scratch/k/k204221/blub/
#
stage_output=`atlas stage --recursive ${src}`
echo ${stage_output}
stage_job_id=`echo ${stage_output} | sed -E 's/^[^0-9]*([0-9]+)[^0-9]*$/\1/g'`

# some initial values
stage_job_state=running
copy_job_state=none
copy_job_id=0
# As long as the staging job is running, we submit copy jobs; when the staging job ended we will start one final iteration of copying
final_copy_job_submitted=false
while [ "$stage_job_state" = "running" ] || [ "$copy_job_state" = "running" ] || [ "${final_copy_job_submitted}" != "true" ]; do
    sleep 300
    if [ "$copy_job_state" = "running" ]; then
        # copy job is still running
        echo "previous copy job ${copy_job_id} still running"
    else
        # copy job not running anymore => restart copy job
        # '--update' makes the copy job skip files which were already
        # retrieved by one of the previous copy jobs
        copy_output=`atlas cp --recursive --update ${src} ${dst}/`
        echo ${copy_output}
        copy_job_id=`echo ${copy_output} | sed -E 's/^[^0-9]*([0-9]+)[^0-9]*$/\1/g'`
        if [ "$stage_job_state" != "running" ]; then
            # When we arrive here, the stage job has ended and a new copy job
            # has been submitted. In this situation we would like to wait until
            # this final copy job is finished and exit the loop afterwards. This
            # is controlled by 'final_copy_job_submitted'
            final_copy_job_submitted="true"
        fi
        # give the new copy job some time to initialize
        sleep 10
    fi

    stage_job_raw_state=`acli jobs get ${stage_job_id} | grep "State:"`
    echo "stage job ${stage_job_id} ${stage_job_raw_state}"
    stage_job_state=`echo ${stage_job_raw_state} | sed 's/^State:[ ]*//g'`
    copy_job_raw_state=`acli jobs get ${copy_job_id} | grep "State:" | sed 's/^State:[ ]*//g'`
    echo "stage job ${stage_job_id} ${stage_job_raw_state}"
    stage_job_state=`echo ${stage_job_raw_state} | sed 's/^State:[ ]*//g'`
    echo "current stage job state is '${stage_job_state}'"
    echo "current copy job state is '${copy_job_state}'"
done

Cache, Online, Offline, Stage, Release!?#

Storage of data on tape is cheap compared to storage on HDD, SSD or similar. Therefore, the DKRZ data archive uses tapes for data storage. However, accessing data from tape is relatively slow compared to other storage types. To overcome this disadvantage, a hierarchical storage management (HSM) system is used at DKRZ which temporarily stores additional copies of archived files on faster storage systems than tape. The new Versity system may store copies of files in three categories of storages. Each storage category can be associated with a time span until requested data are available.

  • primary SSD cache: file instantly available

  • intermediate HDD storage: file accessed delayed by a few seconds

  • tape: file access delayed by a few minutes because file needs to be read from tape

Copies of one file might be stored in multiple storages and storage categories. The file state online, warm or offline indicates where the fastest available copy of the respective file is located:

  • online: 1x file copy in primary SSD cache (possibly also on tape)

  • warm: no file copy in primary SSD cache; 1x file copy in intermediate storage

  • offline: file copies only on tape

The primary cache and the intermediate storage are cleaned up automatically. You can manually tell Versity to remove a file from the cache by releasing it. Commonly, you do not need to do this because the cache is cleaned up automatically quite well.

If you archive a file, it is written into the cache and, later, from there to tape. “Later” in this context can be minutes or hours depending on the load on the system and the availability of tape drives. This process cannot be controlled by you. A file cannot be removed from the cache if it has not been written to tape.

You can stage files as follows:

$ atlas stage <FILE>

# OR

$ atlas stage --recursive <DIRECTORY>

The cleanup of the cache starts when the fill state reaches an upper threshold. Then, the oldest files are removed first – based on the last access time and the last staging time – until the fill state reaches a lower threshold. Thus, the atlas stage command also acts like touch. There is no guarantee that a file remains cached for X hours after running atlas stage. Please do not run the command permanently for all of your files in a loop. If all users were doing this … .

File access via HSM mount points on Levante#

It is planned to make /hsm/arch, /hsm/double and /hsm/doku available via equally named mount points on Levante. The optimal configuration for this setup still has to be clarified and, hence, these mount points are not available from the beginning. We will notify you when they are set up.

File transfer via S3#

If you need to transfer data between the Lustre filesystem and the tape archive, please use ATLAS. However, if data should be transferred between an S3 bucket and the tape archive, the Versity S3 Gateway is best suited.

The archival of data can be done with a S3 client of your choice. Retrieval of files requires a S3 tool which supports the Amazon Glacier command set. Details will be published later on.

File transfer from/to outside DKRZ#

Please contact us via support@dkrz.de

Files excluded from tape storage#

Sometimes temporary files or source code repositories are archived accidentally. They take up unnecessary space on tape and entries in the metadata database. Therefore, starting in August 2026, files with certain names / name-pattern or in folders with certain names will not be written to tape but remain in a separate section of the cache. Owners of these files will be informed in regular intervals and asked to clean up these files.

## all files in certain folders
# any folder starting with '.' in the name
.*\/\..+\/.*
## files with certain extensions
# NOT ALL files starting with "."
.*\/\.[^/]+\.swp$
# <file>.~processed~ (marks <file> as being processed)
.*\.~processed~$
# temporary copy of editor: *.txt~, *.sh~, *.jnl.~<number>~
.*\.txt~$
.*\.sh~$
.*\.jnl\.~[0-9]+~$
# MacOS files "._*" and ".DS_Store" (covered by first file rule)
\._[^/]+$
.DS_Store
# SLURM job output
.*[oe][0-9]{5,8}$
.*\/slurm-[^/]+\.out$
# Python temp folders
.*\/__pycache__\/.*
.*\/[^/]*\.egg-info\/.*

Versity will match a Regular Expression to identify these files. This is it:

# single RexEx:
^.*\/\..+\/.*$
^.*\/\.[^\/]+\.swp$
^.*\.(txt|sh)~$
^.*\.~processed~$
^.*\.jnl\.~[0-9]+~$
^.*[oe][0-9]{5,8}$
^.*\/slurm-[^\/]+\.out$
^.*\/__pycache__\/.*$
^.*\/[^/]*\.egg-info\/.*$

# in one line:
^(.*\/\..+\/.*|.*\/\.[^\/]+\.swp|.*\.(txt|sh)~|.*\.~processed~|.*\.jnl\.~[0-9]+~|.*[oe][0-9]{5,8}|.*\/slurm-[^\/]+\.out|.*\/__pycache__\/.*|.*\/[^/]*\.egg-info\/.*)$
On this page