Getting started#

file version: 30 Jun 2026

Warning

The Versity HSM system is not active yet. We expect it will be online on 6 July 2026.

Overview#

The Versity HSM system is the new software to manage the DKRZ tape archive and replaces StrongLink which ran at DKRZ since November 2021. A serivce called ATLAS is the primary access point for the archival and retrieval of data and most other file operations like listing files or comparing checksums. ATLAS is accessible via

  • a WebGUI

  • a command line interface acli

Running only acli starts an interactive shell whereas running acli <command> just executes the command without entering the interactive shell. The latter mode is meant for scripting. Please use absolute paths in this case.

Important changes are:

  • all paths have the prefix hsm: /hsm/arch, /hsm/double, …

Key features are:

  • automatic packing of small files (invisible to users)

  • async background data transfer via ATLAS: no need for users to run long SLURM job for archivals or retrievals

  • ACLs exist: as file owner, set read/write/execute permissions for individual other users

  • very efficient internal organization of the tape access: no need to sort retrieval requests by tape

  • retention time: set write-protection for user-defined time period

But:

  • If you plan to archive a folder with 60 000 log files of 1 MB size each, please still pack them.

  • Please check whether copy jobs were actually successful before deleting data on work.

  • Files in retention mode cannot be changed at all: no move, no chown. Retention mode can only be released by admins.

Login#

ATLAS is meant as the central tool for data transfer between tape archive and Lustre filesystem. Using ATLAS requires a login with your default DKRZ credentials. On login, a login-token and a refresh-token are generated. The login-token is valid for one hour and the refresh-token for a month. acli automatically creates new login-tokens as long as the refresh token is valid.

$ acli login
Username: k204221
Password:
Logged in as k204221 (role=user, expires in 59m59s)

Check the login status:

$ acli status
Server:     vsy-srv-test-01.hsm.dkrz.de:7449
Node:       vsy-srv-test-01-adm
Version:    v3.4.5-251-g88b18b73
Healthy:    true
Workers:    3
Etcd:       true
Query eng:  true
User:       k204221 (uid=25301)
Role:       user
Expires in: 59m31s
Session:    719h59m31s remaining (~725 refreshes left)

Data transfer#

copy#

Please use the command acli cp for archivals and retrievals. This command starts a copy job in ATLAS and you receive a job id. Thus, it runs asynchronous. It is the same approach as submitting a SLURM job but using another scheduler. ATLAS accesses the Lustre filesystem directly which makes the data transfer much more efficient. Additionally, it does not use network bandwidth of the Levante node on which you start the transfer. These locations are available in ATLAS:

  • /hsm: tape archive; read-write-mode

  • /scratch: Lustre/Levante filesystem, read-write-mode

  • /work: Lustre/Levante filesystem, read-only

For details on the commands upload, download and move please have a look into our Data Transfer documentation page.

Example: archive two files and check job status.

# create directory and start copy process
$ acli mkdir /hsm/arch/ka1209/single_files
$ acli cp /scratch/k/k204221/file_01.txt /work/k/k204221/file_02.txt /hsm/arch/ka1209/single_files/
copy job submitted: 63

# check the job status
$ acli jobs get 63
Job ID:      63
Type:        copy
State:       running
User:        k204221 (uid=25301)
Node:        vsy-srv-test-01-adm
Resource:    scoutfs
Items:       0 / 2
Created:     2026-06-29 10:55:19
Started:     2026-06-29 10:55:20
# job is still running

# retry later:
Job ID:      63
Type:        copy
State:       completed
...
Completed:   2026-06-29 12:55:20
# job is still running

Example: a failed job

# failed job:acli jobs get 64
Job ID:      64
Type:        copy
State:       failed
User:        k204221 (uid=25301)
Node:        vsy-srv-test-01-adm
Source res.: work
Dest res.:   scoutfs
Items:       0 / 1
Errors:      1
Error:       1 of 1 items failed
  /work/testing_2026/archive_export/file_001gb_a.nc: file already exists
Created:     2026-06-29 10:52:01
Started:     2026-06-29 10:52:02
Completed:   2026-06-29 10:52:03

Example: retrieve files from the archive to /scratch.

$ acli cp --recursive /hsm/arch/ka1209/k204221/output_42 /scratch/k/k204221
copy job submitted: 65

Storage location of files#

Versity may store copies of files in three categories of storages. Each storage category can be associated with a time span until requested data are available.

  • primary SSD cache: file instantly available

  • intermediate storage: file accessed delayed by a few seconds

  • tape: file access delayed by a few minutes because file needs to be read from tape

Copies of one file might be stored in multiple storages and storage categories. The file state online, warm or offline indicates where the fastest available copy of the respective file is located:

  • online: 1x file copy in primary SSD cache (possibly also on tape)

  • warm: no file copy in primary SSD cache; 1x file copy in intermediate storage

  • offline: file copies only on tape

As soon as a file is requested by acli cp it is automatically transfered into the primary cache and marked as online. This process is denoted as staging.

The primary cache is cleaned up automatically but files copies cannot be removed from it if not at least one copies of this file has been written to tape.

Listing files#

Files are listed with acli ls:

Print some details with -la as on bash:

$ acli ls /hsm/arch/bm0146 -la
-rw-rw-r--+  k204221  bm0146  9.1M    Jun 15 20:04  aw---  2  acli_3.4.5-20260615060213-8985bbe8_linux_amd64.rpm
-rw-rwxr--+  k204221  bm0146  7B      Jun 21 01:28  ao-r-  2  do_retention.txt
drwxr-xr-x+  k204221  bm0146  16B     Jun 21 01:35  -----     a
drwxr-xr-x+  k204221  bm0146  16B     Jun 21 01:35  -----     b
-rw-rw-r--+  k204221  bm0146  514.6M  Jun 24 08:52  aw---  2  log.tar
-rw-rwxr--+  k204221  bm0146  1.1G    Mar 04 21:31  ao---  2  file_001gb_a.nc
-rw-rwxr--+  k204221  bm0146  1.1G    Mar 04 21:31  ao---  2  testing
-rw-rw-r--+  k204221  bm0146  3.1K    Mar 26 21:37  -o---     install.sh

Compared to bash’s ls, there are two additional columns printed between the date-time and filename columns. The first of these columns consists of five characters (link to full description follows). If the first latter is an a then the file has been savely archived (written to tape). The second latter cotains storage information as described above:

  • second latter is o: online

  • second latter is n: never release from cache => online permanently

  • second latter is w: warm

  • second latter is -: offline

Additional file details are printed with -D and -E:

Here, you find additional time stamps and storage information.

Checksum verification#

Versity verifies each copy process within the HSM system via sha1 checksums – e.g. after staging a file from tape into the cache. The acli cp command does not verify checksums of file copies which are outside of /hsm. The checksums are commonly available via ls -E. But, there is no guarantee that there is one checksum per file.

If you wish to verify your archivals and retrievals based on checksums, please use the checksum command of ATLAS. You have to start the checksum calculation manually and can select the type/algorithm: md5, sha512, … . Checksums can only be calculated and/or verified when a file is online. The checksum command has been made for two verification workflows.

Checksum Workflow A#

  • calculate checksum in Versity

  • obtain checksum

  • compare Versities checksum against local checksum

# calculate checksum in Versity
$ acli checksum --generate --type md5 /hsm/arch/bm0146/install.sh
/hsm/arch/bm0146/install.sh: MD5:75e3e4eb6fb2a55abbd9489e1b9b2725

# calculate checksum locally
$ md5sum install.sh
75e3e4eb6fb2a55abbd9489e1b9b2725  install.sh

Checksum Workflow B#

  • calculate checksum of local file

  • store checksum of local file in Versity

  • ask Versity to verify the file again stored checksum

# calculate checksum locally
$ md5sum install.sh
75e3e4eb6fb2a55abbd9489e1b9b2725  install.sh

# set checksum in Versity
$ acli checksum --set 75e3e4eb6fb2a55abbd9489e1b9b2725 --type md5 /hsm/arch/bm0146/install.sh
/hsm/arch/bm0146/install.sh: md5 checksum set

# verify checksum in Versity
$ acli checksum --verify  /hsm/arch/bm0146/install.sh
/hsm/arch/bm0146/install.sh: OK

The whole process might also fail if a wrong checksum is provided or if a file copy in Versity became damaged.

# set checksum in Versity
$ acli checksum --set 85e3e4eb6fb2a46abbd9489e1b9e2733 --type md5 /hsm/arch/bm0146/install.sh
/hsm/arch/bm0146/install.sh: md5 checksum set

# verify checksum in Versity
$ acli checksum --verify  /hsm/arch/bm0146/install.sh
/hsm/arch/bm0146/install.sh: verify checksum: verify checksum failed: scoutam: verify checksum inode 4896131: rpc error: code = Unknown desc = verify failed: checksum mismatch: 85e3e4eb6fb2a46abbd9489e1b9e2733 != 75e3e4eb6fb2a55abbd9489e1b9b2725
Error: 1 path(s) failed verification

Remove and replace checksums#

Retention#

Versity allows to manually set a retention period per file. Within its retention period, a file cannot be deleted, moved or change its owner. Thus, the retention time can protect a file from being accidentally deleted. The retention time can also be removed manually.

metadata and taggingg#

Work in progress

Python and Versity#

Work in progress

Useful Scripts#

Work in progress

On this page