Getting started#

file version: 9 Jul 2026

Overview#

The Versity HSM system is the new software to manage the DKRZ tape archive and replaces StrongLink, which ran at DKRZ since November 2021. A service called ATLAS is the primary access point for the archival and retrieval of data and most other file operations like listing files or comparing checksums. ATLAS is accessible via

a WebGUI (https://tapearchive.dkrz.de; available from calendar week 29 onwards)
a command line interface atlas

Running only atlas starts an interactive shell whereas running atlas <command> just executes the command without entering the interactive shell. The latter mode is meant for scripting. Please use absolute paths in this case.

Important changes and new features are:

all paths have the prefix /hsm: /hsm/arch, /hsm/double, …
automatic packing of small files (invisible to users); but, 60 000 files of 1 MB each should still be packed in advance
ATLAS can read directly from /work and /scratch (archival). It can write directly to /scratch (retrieval)
async background data transfer via ATLAS: no need for users to run long SLURM job for archivals or retrievals; each transfer has its own job id; please check job status: atlas jobs get <job id>
integrity check of data transfers (copy and move) via sha1 checksums by default
very efficient internal organization of the tape access: no need to sort retrieval requests by tape
Python library: currently under development
ACLs exist: as file owner, set read/write/execute permissions for individual other users
retention time: set write-protection for user-defined time period

But:

If you plan to archive a folder with 60 000 log files of 1 MB size each, please still pack them.
Please check whether copy jobs were actually successful before deleting data on work.
Files in retention mode cannot be changed at all: no move, no chown. Retention mode can only be released by admins.

Load tool#

$ module load tape

Login#

ATLAS is meant as the central tool for data transfer between tape archive and Lustre filesystem. Using ATLAS requires a login with your default DKRZ credentials. On login, a login-token and a refresh-token are generated. The login-token is valid for one hour and the refresh-token for a month. atlas automatically creates new login-tokens as long as the refresh token is valid.

$ atlas login
Username: k204221
Password:
Logged in as k204221 (role=user, expires in 59m59s)

Check the login status:

$ atlas status
Server:     vsy-srv-test-01.hsm.dkrz.de:7449
Node:       vsy-srv-test-01-adm
Version:    v3.4.5-251-g88b18b73
Healthy:    true
Workers:    3
Etcd:       true
Query eng:  true
User:       k204221 (uid=25301)
Role:       user
Expires in: 59m31s
Session:    719h59m31s remaining (~725 refreshes left)

Data transfer#

copy#

Please use the command atlas cp for archivals and retrievals. This command starts a copy job in ATLAS and you receive a job id. Thus, it runs asynchronously. It is the same approach as submitting a SLURM job but using another scheduler. Copy jobs target only folders and regular files. Symbolic links are ignored without anywarning. ATLAS accesses the Lustre filesystem directly which makes the data transfer much more efficient. Additionally, it does not use network bandwidth of the Levante node on which you start the transfer. These locations are available in ATLAS:

/hsm: tape archive; read-write-mode
/scratch: Lustre/Levante filesystem, read-write-mode
/work: Lustre/Levante filesystem, read-only

Until next major release end of August: If you copy data from /hsm to the Lustre filesystem, only cached files can be transfered. If not all files are cached, they need to be staged as shown in the example below. We provide a template for combining ‘stage’ and ‘cp’ in larger retrievals.

For details on the commands upload, download and move please have a look into our Data Transfer documentation page.

Example: archive two files and check job status.

# create directory, stage data and start copy process
$ atlas mkdir /hsm/arch/ka1209/single_files
$ atlas stage /scratch/k/k204221/file_01.txt /work/k/k204221/file_02.txt
staging job submitted: 62
$ # wait a bit / check job state

$ atlas cp /scratch/k/k204221/file_01.txt /work/k/k204221/file_02.txt /hsm/arch/ka1209/single_files/
copy job submitted: 63

# check the job status
$ atlas jobs get 63
Job ID:      63
Type:        copy
State:       running
User:        k204221 (uid=25301)
Node:        vsy-srv-test-01-adm
Resource:    scoutfs
Items:       0 / 2
Created:     2026-06-29 10:55:19
Started:     2026-06-29 10:55:20
# job is still running

# retry later:
Job ID:      63
Type:        copy
State:       completed
...
Completed:   2026-06-29 12:55:20
# job is still running

Example: a failed job

# failed job:atlas jobs get 64
Job ID:      64
Type:        copy
State:       failed
User:        k204221 (uid=25301)
Node:        vsy-srv-test-01-adm
Source res.: work
Dest res.:   scoutfs
Items:       0 / 1
Errors:      1
Error:       1 of 1 items failed
  /work/testing_2026/archive_export/file_001gb_a.nc: file already exists
Created:     2026-06-29 10:52:01
Started:     2026-06-29 10:52:02
Completed:   2026-06-29 10:52:03

Example: retrieve files from the archive to /scratch.

$ atlas stage --recursive /hsm/arch/ka1209/k204221/output_42
staging job submitted: 65
$ # wait a bit / check job state
$ atlas cp --recursive /hsm/arch/ka1209/k204221/output_42 /scratch/k/k204221
copy job submitted: 66

Storage location of files#

Versity may store copies of files in three categories of storages. Each storage category can be associated with a time span until requested data are available.

primary SSD cache: file instantly available
intermediate storage: file accessed delayed by a few seconds
tape: file access delayed by a few minutes because file needs to be read from tape

Copies of one file might be stored in multiple storages and storage categories. The file state online, warm or offline indicates where the fastest available copy of the respective file is located:

online: 1x file copy in primary SSD cache (possibly also on tape)
warm: no file copy in primary SSD cache; 1x file copy in intermediate storage
offline: file copies only on tape

Only online files can be served to you. Currently, you have to request the transfer of a file into the primary cache manually. This process is denoted as staging. In future, as soon as a file is requested by atlas cp it is automatically transferred into the primary cache.

The primary cache is cleaned up automatically. In this context, there is no danger of losing new data that has not been written to tape yet: files’ copies cannot be removed from the cache if at least one copy of this file has been written to tape.

But, how to see whether a file is cached? Is there a isCache command? No, but atlas ls -la … provides you with all information you need (next section)!

Listing files#

Files are listed with atlas ls:

$ atlas ls /hsm/arch/bm0146
atlas_3.4.5-20260615060213-8985bbe8_linux_amd64.rpm
do_retention.txt
a
b
log.tar
file_001gb_a.nc
testing

Print some details with -la as on bash:

$ atlas ls /hsm/arch/bm0146 -la
-rw-rw-r--+  k204221  bm0146  9.1M    Jun 15 20:04  aw---  2  atlas_3.4.5-20260615060213-8985bbe8_linux_amd64.rpm
-rw-rwxr--+  k204221  bm0146  7B      Jun 21 01:28  ao-r-  2  do_retention.txt
drwxr-xr-x+  k204221  bm0146  16B     Jun 21 01:35  -----     a
drwxr-xr-x+  k204221  bm0146  16B     Jun 21 01:35  -----     b
-rw-rw-r--+  k204221  bm0146  514.6M  Jun 24 08:52  aw---  2  log.tar
-rw-rwxr--+  k204221  bm0146  1.1G    Mar 04 21:31  ao---  2  file_001gb_a.nc
-rwxr-xr-x+  k204221  ka1209  186.9G  Dec 05 10:25  a-s--  2  iow_data3_001.tar
-rw-rwxr--+  k204221  bm0146  1.1G    Mar 04 21:31  ao---  2  testing
-rw-rw-r--+  k204221  bm0146  3.1K    Mar 26 21:37  -o---     install.sh

Compared to bash’s ls, there are two additional columns printed between the date-time and filename columns. The first of these columns consists of five characters (full description). If the first letter is an a then the file has been safely archived (written to tape). The second letter contains storage information as described above:

second letter is o: online
second letter is n: never release from cache => online permanently
second letter is w: warm
second letter is p: partly online (files internally split into multiple parts)
second letter is -: offline

Additional file details are printed with -D and -E:

$ atlas ls /hsm/arch/bm0146/file_001gb_a.nc -D
file_001gb_a.nc:
  mode: -rw-rwxr--+  links:   1  owner: k204221  group: bm0146
  size: 1200013848  (4k)blocks: 292973/0  inode: 4896129
  archdone;
  ext   : ---- 2026-06-29 13:01:55 0x1557.0x0 - POSIX dkec d21/f87
  copy 1: ---- 2026-06-29 13:02:55 0x10c2.0x0 - POSIX dkar d16/f194
  data version: 292973
  atime: 2026-06-29 11:02:53    mtime: 2026-03-04 21:31:39
  ctime: 2026-06-29 11:02:55  restime: 2026-03-04 22:31:39
  crtime: 2026-06-29 12:55:06 rtntime: -
  uuid: 09e61706-338e-409d-b8ec-c1d6dc140b95

Here, you find additional time stamps and storage information.

Checksum verification#

The atlas cp command verifies file transfers automatically by calculating sha1 checksums. If the verification of a file copy fails, the respective copy is deleted and the transfer is retried. The number of retries can be set via an argument. The checksums are commonly available via atlas ls -E.

If you wish to verify your archivals and retrievals based on checksums, please use the checksum command of ATLAS. You have to start the checksum calculation manually and can select the type/algorithm: md5, sha512, … . Checksums can only be calculated and/or verified when a file is online. The checksum command has been made for two verification workflows.

Note

There is one metadata field per file available to store a checksum. If a sha1 checksum is attached to a file and you request a md5 checksum, then the sha1 checksum will be removed. In order to keep multiple checksums per file, please store them in tags.

Versity verifies each copy process within the HSM system via sha1 checksums – e.g. after staging a file from tape into the cache. These checksums are not visible to users and stored independently of the user checksum.

Checksum Workflow A#

calculate checksum in Versity
obtain checksum
compare Versities checksum against local checksum

# calculate checksum in Versity
$ atlas checksum --generate --type md5 /hsm/arch/bm0146/install.sh
/hsm/arch/bm0146/install.sh: MD5:75e3e4eb6fb2a55abbd9489e1b9b2725

# calculate checksum locally
$ md5sum install.sh
75e3e4eb6fb2a55abbd9489e1b9b2725  install.sh

Checksum Workflow B#

calculate checksum of local file
store checksum of local file in Versity
ask Versity to verify the file again stored checksum

# calculate checksum locally
$ md5sum install.sh
75e3e4eb6fb2a55abbd9489e1b9b2725  install.sh

# set checksum in Versity
$ atlas checksum --set 75e3e4eb6fb2a55abbd9489e1b9b2725 --type md5 /hsm/arch/bm0146/install.sh
/hsm/arch/bm0146/install.sh: md5 checksum set

# verify checksum in Versity
$ atlas checksum --verify  /hsm/arch/bm0146/install.sh
/hsm/arch/bm0146/install.sh: OK

The whole process might also fail if a wrong checksum is provided or if a file copy in Versity became damaged.

# set checksum in Versity
$ atlas checksum --set 85e3e4eb6fb2a46abbd9489e1b9e2733 --type md5 /hsm/arch/bm0146/install.sh
/hsm/arch/bm0146/install.sh: md5 checksum set

# verify checksum in Versity
$ atlas checksum --verify  /hsm/arch/bm0146/install.sh
/hsm/arch/bm0146/install.sh: verify checksum: verify checksum failed: scoutam: verify checksum inode 4896131: rpc error: code = Unknown desc = verify failed: checksum mismatch: 85e3e4eb6fb2a46abbd9489e1b9e2733 != 75e3e4eb6fb2a55abbd9489e1b9b2725
Error: 1 path(s) failed verification

Remove and replace checksums#

# newly generate a checksum
$ atlas checksum --generate --type md5  /hsm/arch/bm0146/install.sh
/hsm/arch/bm0146/install.sh: MD5:75e3e4eb6fb2a55abbd9489e1b9b2725

# remove a stored checksum
$ atlas checksum -c  /hsm/arch/bm0146/install.sh
/hsm/arch/bm0146/install.sh: checksum cleared

Retention#

Versity allows to manually set a retention period per file. Within its retention period, a file cannot be deleted, moved or change its owner. Thus, the retention time can protect a file from being accidentally deleted. The retention time can also be removed manually.

# set retention time
$ atlas retention /hsm/arch/bm0146/install.sh -p 20s
Retention set on 1 file(s)

# try to remove file in retention period
$ atlas rm /hsm/arch/bm0146/install.sh
Error: remove: rpc error: code = PermissionDenied desc = remove /mnt/hsm/arch/bm0146/install.sh: operation not permitted
# fails!

# check how long the retention period last
$ atlas retention /hsm/arch/bm0146/install.sh
/hsm/arch/bm0146/install.sh: 2026-06-29T20:27:04+02:00

# wait 30 seconds and check the retention time
sleep 30
$ atlas retention /hsm/arch/bm0146/install.sh
/hsm/arch/bm0146/install.sh: none

metadata and tagging#

Writing a short summary in progress. Please have a look into the long doc pages linked in the menu on the left.

Files excluded from tape storage#

Sometimes temporary files or source code repositories are archived accidentally. They take up unnecessary space on tape and entries in the metadata database. Therefore, starting in August 2026, files with certain names / name-pattern or in folders with certain names will not be written to tape but remain in a separate section of the cache. Owners of these files will be informed in regular intervals and asked to clean up these files.

## all files in certain folders
# any folder starting with '.' in the name
.*\/\..+\/.*
## files with certain extensions
# NOT ALL files starting with "."
.*\/\.[^/]+\.swp$
# <file>.~processed~ (marks <file> as being processed)
.*\.~processed~$
# temporary copy of editor: *.txt~, *.sh~, *.jnl.~<number>~
.*\.txt~$
.*\.sh~$
.*\.jnl\.~[0-9]+~$
# MacOS files "._*" and ".DS_Store" (covered by first file rule)
\._[^/]+$
.DS_Store
# SLURM job output
.*[oe][0-9]{5,8}$
.*\/slurm-[^/]+\.out$
# Python temp folders
.*\/__pycache__\/.*
.*\/[^/]*\.egg-info\/.*

Versity will match a Regular Expression to identify these files. This is it:

# single RexEx:
^.*\/\..+\/.*$
^.*\/\.[^\/]+\.swp$
^.*\.(txt|sh)~$
^.*\.~processed~$
^.*\.jnl\.~[0-9]+~$
^.*[oe][0-9]{5,8}$
^.*\/slurm-[^\/]+\.out$
^.*\/__pycache__\/.*$
^.*\/[^/]*\.egg-info\/.*$

# in one line:
^(.*\/\..+\/.*|.*\/\.[^\/]+\.swp|.*\.(txt|sh)~|.*\.~processed~|.*\.jnl\.~[0-9]+~|.*[oe][0-9]{5,8}|.*\/slurm-[^\/]+\.out|.*\/__pycache__\/.*|.*\/[^/]*\.egg-info\/.*)$

Python and Versity#

Work in progress

Useful Scripts#

Work in progress