DKRZ HSM (tape archive)#

file version: 05 Feb 2025

Introduction#

The DKRZ operates a hierarchical storage management (HSM) system for medium-to-long term storage of large volumes of data. We offer self-service archival and retrieval for medium-term storage and curated long-term storage via DOKU or WDCC. Storage quota for self-service archival and for DOKU is provided on project-basis and not per user. The storage period for the self-service archival is the project lifetime plus one year. Details on curated long term archival are provided in Data Services -> Archiving & Preserving.

The software installed to operate the HSM is StrongLink. All self-service user interactions with the tape archive are facilitated by command line tools slk and slk_helpers and additional wrapper scripts provided by us. Files archived via DOKU can also be retrieved via slk / slk_helpers.

The hardware of the HSM consists of two spatially separated tape archives. The primary tape archive is located in the DKRZ building in Hamburg and consists of a 2.5 PB hard disk cache and multiple tape libraries providing more than 300 PB storage capacity. All archived files are stored here. The secondary tape archive is located at the Max Planck Computing and Data Facility (MPCDF) in Garching. All files archived via curated long-term storage are duplicated to Garching. We offer two namespaces for self-service archival: one with duplication to Garching and twofold quota allocation; one without duplication to Garching.

If you have questions, which are not answered in this documentation, please contact us via support@dkrz.de.

Storage options, quota and file size#

The tape archive delivers its best performance if the files are sufficiently large. The transfer speed of files larger than approximately 250 GB between Tape and HSM cache decreases due to cache limitations. Therefore, we recommend to archive files in the size range from 10 GB to 200 GB. We account at least 1 GB quota per archived file and recommend to pack small files. You can use packems to pack small files into tar balls and create indices automatically. Archivals of files larger than 500 GB have be tested successfully.

The amount of data that can be stored in the tape archive per project is limited by the available storage quota of that project. Individual users do not have a quota. There is normal tape archive quota for self-service archival and quota for curated long-term archival DOKU. Additionally, data can be long-term archived via the WDCC. The default root namespace for self-service is /arch/<project>. Files archived to this namespace are stored on one tape in Hamburg. Instead, users can archive very important files to /double/<project> which will then be stored twice, i.e. one copy in Hamburg and one copy in Garching. Long-term archival data is stored twice by default.

Storage space on the HSM is applied for in conjunction with the (bi-)annual application for DKRZ compute and storage resources. You can check your storage quota via https://luv.dkrz.de. The quota Archive Project denotes quota for self-service archival and Archive Long Term denotes quota for curated DOKU archivals.

The following table provides an overview:

Storage Type

root namespace

Storage period

copy in Garching

quota

self-service

/arch/<prj>

project period plus 1 year

no

Archive Project

/double/<prj>

yes

curated

DOKU

/doku/<prj>

project period plus 10 years

yes

Archive Long- Term

WDCC

-

> 10 years

command line tools for tape access#

slk and slk_helpers#

The command line tool slk and its add-on slk_helpers allows users to interact with the HSM. The slk is the official StrongLink command line tool. It is lacking some useful features and its usage in scripts is limited. Therefore, the DKRZ developed slk_helpers as an add-on. The slk_helpers can be extended on user request. On Levante HPC, both tools are installed system-wide and accessible to all users via module load slk.

Further reading:

The functionality of the slk data retrieval command is limited on the Levante login nodes. It should be used on shared and interactive nodes.

Note

slk stores a login token in the home directory of each user (~/.slk/config.json). The login token is valid for 30 days. By default, this file can only accessed by the respective user (permissions: -rw------- / 600 ). However, users should be careful when doing things like chmod 755 * in their home directory. If you assume that your slk login token has been compromised please contact support@dkrz.de .

SLURM job wrapper scripts#

DKRZ provides SLURM job wrapper scripts that facilitate certain StrongLink tasks which require waiting time or should run in the background. These scripts combine slk and slk_helpers commands and submit one or multiple SLURM jobs. They are loaded together with slk and slk_helpers via the module slk.

Further reading:

packems: packing of data#

The tape archive delivers its best performance if the archived files are in the size range of 10 GB to 250 GB. Therefore, small files should be packed into tar balls prior archiving. For this purpose, MPI-M and DKRZ developed packems. packems depends on slk, the slk_helpers and pyslk (see below).

The Users can provide a list of files to packems. packems then automatically distributes them into tar balls with a targeted size of 100 Gb. It also creates an index file which contains information about which tar ball each individual file is located in. Please use module load packems to load the packems tool on HPC Levante.

StrongLink automatically imports metadata from netCDF files into DKRZ-internal StrongLink metadata database (see below). This StrongLink feature does not work with packed files. Hence, users need to weigh the advantages of packing against the automatic metadata import.

Further Reading: * basic usage example * extended usage examples * packems manual

pyslk: python slk wrapper#

We offer a Python library for interacting with the tape archive. Technically, the library consists of wrappers around slk and the slk_helpers. Therefore, it requires an environment where slk and slk_helpers are installed.

Further reading:

External access#

The tape archive can only be accessed via Levante.

Backend data handling#

Just like with the previous HSM system HPSS, the fast disk cache is installed upstream of the tape system. Files selected for archival are first copied to the disk cache and then successively written onto tape. Files selected for retrieval are first copied from tape to the cache and then copied to the specified target locations. The retrieval of files that are still/already stored in the disk cache is considerably faster than the retrieval of files that are located on tape only.

The distribution of the files in the disk cache, primary tape archive and secondary tape archive is automatically controlled by the software StrongLink.

A - or t is appended to the permissions string of each file in the output of slk list and slk_helpers search_list. The - indicates that the file is stored in the cache. The t indicates the file is stored on tape. Alternatively, the command slk_helpers iscached can be used to check whether a file is currently stored in the cache.

Further reading#

On this page