Versity HSM System (tape archive)#

file version: 08 Oct 2026

Introduction#

The DKRZ operates a hierarchical storage management (HSM) system for medium- to long-term storage of large volumes of data. We offer self-service archival and retrieval for medium-term storage, as well as curated long-term storage via DOKU or WDCC. Storage quotas for self-service archival and DOKU are allocated on a project basis rather than per user. Data stored through the self-service archival service is retained for the duration of the project plus one additional year. Details on curated long term archival are available under Data Services -> Archiving & Preserving.

The software used to operate the HSM is ScoutAM by Versity. All self-service user interactions with the tape archive are facilitated by ATLAS which offers both a command line interface and a web UI. Additionally, read-only mount points are provided on all HPC nodes. Files archived via DOKU can also be accessed via ATLAS and via these mount points.

The hardware supporting the HSM consists of two geographically separated tape archives. The primary tape archive is located in the DKRZ building in Hamburg, where all archived files are stored. The secondary tape archive is located at the Max Planck Computing and Data Facility (MPCDF) in Garching. All files archived via curated long-term storage are duplicated to Garching. We offer two namespaces for self-service archival: one with duplication to Garching and twofold quota allocation, and one without duplication to Garching.

If you have questions that are not answered in this documentation, please contact us via support@dkrz.de.

Storage options, quota and file size#

The tape archive delivers its best performance if the files are sufficiently large but not too large. Please consider a file size of 300 GB as a soft upper limit. There is no lower limit for the file size because the Versity HSM system automatically combines small files in packs. However, we recommend to pack files manually if more than 1000 files of less than 10 MB to 100 MB average size are archived. packems might be helpful.

The amount of data that can be stored in the tape archive per project is limited by the available storage quota of that project. Individual users do not have a quota. There is normal tape archive quota for self-service archival and a separate quota for curated long-term archival (DOKU). Additionally, data can be long-term archived via the WDCC. The default root namespace for self-service is /hsm/arch/<project>. Files archived to this namespace are stored on one tape in Hamburg. Instead, users can archive very important files to /hsm/double/<project> which will then be stored twice, i.e. one copy in Hamburg and one copy in Garching. Long-term archival data is stored twice by default.

Currently, at least 1 GB is accounted for each archived file. This was introduced in HPSS and kept in StrongLink in order to incentivice packing of small files. In future, this minimum accounted quota per file will be reduced to a lower value or replaced by a file number quota. Which of both will come is not decided yet.

Storage space on the HSM is applied for in conjunction with the (bi-)annual application for DKRZ compute and storage resources. You can check your storage quota via https://luv.dkrz.de. The quota Archive Project denotes the quota for self-service archival and Archive Long Term denotes the quota for curated DOKU archival.

The following table provides an overview:

Storage Type

Root namespace

Storage period

Copy in Garching

Quota

Self-service

/hsm/arch/<project>

project period plus 1 year

no

Archive Project

Self-service

/hsm/double/<project>

project period plus 1 year

yes

Archive Project

Curated (DOKU)

/hsm/doku/<project>

project period plus 10 yrs

yes

Archive Long-Term

WDCC

> 10 years

(see WDCC)

(see WDCC)

Sometimes temporary files or source code repositories are archived accidentally. Starting in August 2026, files with certain names / name-pattern or in folders with certain names will not be written to tape but remain in a separate section of the cache. Owners of these files will be informed in regular intervals. Details are given here.

Tools and Interfaces#

The interface provided to users for the self-service archival and retrieval is called ATLAS. There are two access methods:

packing of data#

The HSM software by Versity is configured to pack small files automatically. This process is not visible to users and cannot be controlled by users. By default, files are packed in the order in which they are transferred into the HSM.

Therefore, there is no general recommendation to pack small files as with the previous two systems – HPSS and StrongLink. Nevertheless, if you wish to archive more than approximately 1000 files of less than 10 MB to 100 MB average size we recommend to pack them in advance. This procedure can improve the retrieval speed because the automatic packing feature does not know which files belong together. Packing files in advance is particularly valuable, if you archive a TB-size or PB-size dataset where you already know which subsets will be retrieved together.

Using packems simplifies the packing process considerably for you.

If you wish support in packing and archiving data, please write us (support@dkrz.de).

Data access via Python#

In preparation.

External access#

In preparation.

On this page