HSM tools: June 2025 updates#

New versions of the slk_helpers (1.16.5) and of the slk_wrappers (2.4.0) are available which contain several new features and bugfixes. The major changes focuses on improvements of the retrieval workflow. Additionally, an experimental version of a command called slk_helpers resource_features is released which can print various resource details such as caching state, size and various time stamps in a CSV-like way. An exact list of changes since the last officially announced release can be found in the changelog blog entry.

New retrieval workflow: further improvements#

In Februar, we announced the new recall and retrieval scripts called recall watcher and retrieve watcher. Apart from fixing a few bugs, we simplified the process of starting and stopping these scripts. To do so, the slk_helpers received three new commands:

  • slk_helpers init_watchers: generates various files required by the new scripts (wrapper to slk_helpers gfbt reducing number of arguments; details)

  • slk_helpers start_watchers: submit recall and retrieve SLURM job scripts

  • slk_helpers stop_watchers: cancel recall and retrieval SLURM jobs

If you wish to retrieve <a list of archived files> to <a destination location>, please do as follows:

mkdir -p <tmp folder>
cd <tmp folder>
module load slk
slk_helpers init_watchers <a list of archived files> <additional parameters> -d <a destination location> -ns
slk_helpers start_watchers <DKRZ project slurmJobAccount>

Two types of SLURM job scripts are submitted which recall (transfer: from tape to cache) and retrieve (transfer: from cache to you/Lustre) the requested archived files. The SLURM job scripts submit themselves repeatedly until all files have been transfered back. Check the retrieve.log and recall.log files to receive status information of the running process. If you wish to cancel the process, please change into the <tmp folder> and run

slk_helpers stop_watchers

Both start_watchers and stop_watchers offer arguments to start only the recall (--recall-watcher-only) or only the retrieve watcher (--retrieve-watcher-only).

New command: resource_features#

The slk_helpers contain multiple commands to access individual resource information such size, caching state or on which tape(s) a resource is stored on. This set of commands grew historically. The command usage and output structure differs slightly, which is confusing – particularly for new users. In an attempt to unify the access on various resource details, the command resource_features was introduced. The command is considered to be experimental. The command itself and its arguments might be renamed, removed or kept anytime without prior notice. Anyway, we would be very interested in your feedback on this new command.

If you wish to print path, size and caching state of all four files in /arch/bm0146/k204221/iow, please do this:

$ slk_helpers resource_features /arch/bm0146/k204221/iow --features path,size,cached
id,path,size,cached
49058705519,/arch/bm0146/k204221/iow/iow_data_006.tar,8364490752,no
49058705518,/arch/bm0146/k204221/iow/iow_data_005.tar,20478689280,no
49058705517,/arch/bm0146/k204221/iow/iow_data_004.tar,20715667456,no
49058705497,/arch/bm0146/k204221/iow/INDEX.txt,1268945,yes
Resources: 4; Resources with Errors: 0

Usage details and details on the output format are given on the slk_helper manual page.

Changelog slk_helpers v1.16.5  
On this page