DKRZ HSM (tape archive)#

file version: 17 May 2024

Introduction#

The DKRZ operates a hierarchical storage management (HSM) system for medium-to-long term storage of large volumes of data.

The hardware of the HSM consists of a disk cache and two tape archives. The primary tape archive has a storage capacity of more than 300 PB and is located in the DKRZ building in Hamburg. Selected files are mirrored to the secondary tape archive located at the Max Planck Computing and Data Facility (MPCDF) in Garching. The software installed to operate the HSM is StrongLink. All command line based interactions with the tape archive are facilitated by StrongLink and its command line tool slk.

If you have questions, which are not answered in this documentation, please contact us via support@dkrz.de.

Storage options, quota and file size#

The tape archive delivers its best performance if the files are sufficiently large. The transfer speed of files larger than approximately 250 GB between Tape and HSM cache is lower due to cache limitations. Therefore, we recommend to archive files in the size range from 10 GB to 200 GB. The accounting of used quota is done in increments of 1GB per archived file. You can use packems to pack small files into tar balls and create indices automatically. Archivals of files larger than 500 GB have be tested successfully.

The amount of data that can be stored in the tape archive per project is limited by the available storage quota of that project. Individual users do not have a quota. Storage space on the HSM is applied for in conjunction with the (bi-)annual application for DKRZ compute and storage resources. You can check your storage quota via https://luv.dkrz.de . There is normal tape archive quota denoted as arch and quota for long term archival denoted as docu. Additionally, users might select very important files to be stored twice, i.e. one copy in Hamburg and one copy in Garching. The following table provides an overview:

Storage location, time and quota#

File Storage

Storage Time

Used quota

How to achieve this

Past location (HPSS)

New location (StrongLink)

single copy on tape

1 year after expiration of DKRZ project

arch quota

default storage type

/hpss/arch/<prj>

/arch/<prj>

second copy on separate tape

1 year after expiration of project

arch quota; double file size used

store data in specific root namespace (see right column)

/hpss/double/<prj>

/double/<prj>

long-term storage for reference purpose

10 years after expiration of project

docu quota

Please contact data@dkrz.de

/hpss/doku/<prj>

/doku/<prj>

command line tools for tape access#

slk and slk_helpers#

The command line tool slk and its add-on slk_helpers allows users to interact with the HSM. The slk is the official StrongLink command line tool. It is lacking some useful features and its usage in scripts is limited. Therefore, the DKRZ developed slk_helpers as an add-on. The slk_helpers can be extended on user request. On Levante HPC, both tools are installed system-wide and accessible to all users via module load slk.

Further reading:

The functionality of the slk data retrieval command is limited on the Levante login nodes. It should be used on shared and interactive nodes.

Note

slk stores a login token in the home directory of each user (~/.slk/config.json). The login token is valid for 30 days. By default, this file can only accessed by the respective user (permissions: -rw-------/600). However, users should be careful when doing things like chmod 755 * in their home directory. If you assume that your slk login token has been compromised please contact support@dkrz.de .

The StrongLink software comes with a command line tool suite slk. slk is the user interface to the StrongLink software and allows the user to interact with the HSM. The available commands are:

Note

StrongLink uses the term “namespace” or “global namespace” (gns). A “namespace” is comparable to a “directory” or “path” on a common file system.

Note

slk does not provide its own shell. slk login simply creates a login token which allows other slk commands to be used. It is not possible to navigate through the emulated directory structure of the HSM using a cd-like command. Instead, each slk commands needs the full path of resources as input.

Note

On Levante login nodes, slk retrieve can only retrieve one file at once. There are no limitations for retrievals on nodes of the compute, shared and interactive partitions. All other slk commands are available without any limitations on all nodes. If you wish to archive or retrieve large files or many files interactively, please start an interactive batch session via the interactive partition with salloc (Run slk in the “interactive” partition and Data Processing on Levante).

Please read Known Issues before you start using the slk the first time. Please have a look into StrongLink Command Line Interface (slk) (on docs.dkrz.de) or into the StrongLink Command Line Interface Guide for a detailed description of the individual commands. Alternatively, the page slk Usage Examples contains several usage examples.

slk helpers: add-on to slk#

The slk is lacking a few minor but very useful features. The slk_helpers program adds these features. Its usage is very similar to the usage of slk. The following commands are available:

Please have a look into slk helpers (on docs.dkrz.de) for a detailed description of the individual commands. Alternatively, the section slk Usage Examples contains several usage examples.

SLURM job wrapper scripts#

DKRZ provides SLURM job wrapper scripts that facilitate certain StrongLink tasks which require waiting time or should run in the background. These scripts combine slk and slk_helpers commands and submit one or multiple SLURM jobs. They are loaded together with slk and slk_helpers via the module slk.

Further reading:

packems: packing of data#

The tape archive delivers its best performance if the archived files are in the size range of 10 GB to 250 GB. Therefore, small files should be packed into tar balls prior archiving. For this purpose, MPI-M and DKRZ developed packems. packems depends on slk, the slk_helpers and pyslk (see below).

The Users can provide a list of files to packems. packems then automatically distributes them into tar balls with a targeted size of 100 Gb. It also creates an index file which contains information about which tar ball each individual file is located in. Please use module load packems to load the packems tool on HPC Levante.

StrongLink automatically imports metadata from netCDF files into DKRZ-internal StrongLink metadata database (see below). This StrongLink feature does not work with packed files. Hence, users need to weigh the advantages of packing against the automatic metadata import.

Further Reading: * basic usage example * packems manual

pyslk: python slk wrapper#

We offer a Python library for interacting with the tape archive. Technically, the library consists of wrappers around slk and the slk_helpers. Therefore, it requires an environment where slk and slk_helpers are installed.

Further reading:

Packing of data#

External access via sftp#

The tape archive can only be accessed via Levante.

Backend data handling#

Just like with the previous HSM system HPSS, the fast disk cache is installed upstream of the tape system. Files selected for archival are first copied to the disk cache and then successively written onto tape. Files selected for retrieval are first copied from tape to the cache and then copied to the specified target locations. The retrieval of files that are still/already stored in the disk cache is considerably faster than the retrieval of files that are located on tape only.

The distribution of the files in the disk cache, primary tape archive and secondary tape archive is automatically controlled by the software StrongLink.

A - or t is appended to the permissions string of each file in the output of slk list and slk_helpers search_list. The - indicates that the file is stored in the cache. The t indicates the file is stored on tape. Alternatively, the command slk_helpers iscached can be used to check whether a file is currently stored in the cache.

Further reading#