Retrievals from tape

file version: 25 August 2022

current software versions: slk version 3.3.21; slk_helpers version 1.2.4

Introduction and Summary

The slk retrieve command is not available on the login nodes of Levante but only via the partitions compute, shared and interactive. If you wish to use slk retrieve interactively, please start an interactive batch session via the interactive partition with salloc (Data Processing on Levante). The partition slk, on which slk retrieve works as well, will be removed in the beginning of June.

Note

Please note the section Aggregate file retrievals in order to speed up your retrievals.

Warning

The permissions of all retrieved files are rw------- and have to be adapted manually. Using umask and setfacl in advance does not work.

Recommendations for usage of slk retrieve

• High memory usage: Please allocate 6 GB of memory for each call of slk retrieve (argument for sbatch: --mem=6GB). Otherwise, your commands might be killed by the operating system. If you plan to run three retrievals in parallel, please allocate 18 GB – and so on.

• On Levante, recommended: Please striped the target folder of retrievals as follows lfs setstripe -E 1G -c 1 -S 1M -E 4G -c 4 -S 1M -E -1 -c 8 -S 1M TARGET_FOLDER.

• Check exit codes: Information on the success or failure of slk retrieve will not be printed into the SLURM log automatically. We strongly suggest to check the exit code of each slk retrieve and print it to the job log. The variable $? holds the exit code of the preceding command (see How do I capture exit codes?; example: our example scripts for usage examples).

• Please also be aware that some issues on Levante might cause slk to crash randomly (see section slk issues on Levante on page Known Issues).

Speed up your retrievals

When you retrieve data from the HSM, these data are first copied from a tape to the HSM cache and, in a second step, copied from the HSM-cache to your _local_ file system (Lustre). If the data are already located in the cache, the first step is automatically omitted. Commonly, copying a file from tape to cache takes longer than copying it from cache to the Lustre file system. Therefore, it is good to know where the needed files are currently stored in order to estimate the time needed for retrieval. Below some hints how to improve the speed of your file retrievals.

Is a file stored in the HSM cache and/or on tape?

The output of slk list indicates whether a file is stored in the HSM cache or not. If the 11th character of the permissions-string is a t then the file is stored exclusively on tape. If it is a - then the file is stored in the cache. In the latter case, the user does not know whether the file is additionally stored on tape or not, for example if the file was archived shortly before slk list was performed and had not yet been transferred to tape. Example:

$ slk list /arch/ex/am/ple
-rw-r--r---   k204221   bm0146  11 B    02 Mar 2021     file_1.txt
-rw-r--r--t   k204221   bm0146  16 B    02 Mar 2021     file_2.txt
-rw-r--r--t   k204221   bm0146  15 B    02 Mar 2021     file_3.txt

Example explained: The file file_1.txt is stored in the cache and can be quickly retrieved. The files file_2.txt and file_3.txt are only stored on tape and their retrieval will take more time.

Additionally, the slk_helpers feature a command iscached, which prints out information on the location of storage. Please note that the exit code of this command is 1 if the tested file is not cached (see How do I capture exit codes?). Example:

$ slk_helpers iscached /arch/ex/am/ple/file_2.txt
File is not cached
$ echo $?
1

$ slk_helpers iscached /arch/ex/am/ple/file_1.txt
File is cached
$ echo $?
0

Aggregate file retrievals

When several files shall be retrieved, it is most efficient to retrieve all files at once with one call of slk retrieve instead of retrieving each file with an individual call of slk retrieve. First, all files that are stored on one tape will be read from that tape at once. Individual slk retrieve calls would cause the tape to be ejected and brought back to its shelf after each retrieval. Second, one slk retrieve command might use several tape drives to copy files from distinct tapes in parallel. Using one slk retrieve call per file does not allow using this feature. Therefore, it is useful to aggregate several file retrievals into one.

Recursive retrievals

When an entire namespace or most files of one namespace shall be retrieved, this namespace should be retrieved via one recursive slk retrieve call. Example

$ slk list /ex/am/ple/data | cat
-rw-r--r--- k204221     bm0146          1.2M   10 Jun 2020  INDEX.txt
-rw-r--r--t k204221     bm0146         19.5G   05 Jun 2020  out_data_001.tar
-rw-r--r--t k204221     bm0146         19.0G   05 Jun 2020  out_data_002.tar
-rw-r--r--t k204221     bm0146         19.4G   05 Jun 2020  out_data_003.tar
-rw-r--r--t k204221     bm0146         19.3G   05 Jun 2020  out_data_004.tar
-rw-r--r--t k204221     bm0146         19.1G   05 Jun 2020  out_data_005.tar
-rw-r--r--t k204221     bm0146          7.8G   05 Jun 2020  out_data_006.tar
Files: 7

$ slk retrieve -R /ex/am/ple/data /tar/get/folder
[                                        ] 100% complete. Files retrieved: 7/7, [105.3G/105.3G].

$ ls /tar/get/folder
INDEX.txt    out_data_001.tar    out_data_002.tar    out_data_003.tar    out_data_004.tar    out_data_005.tar    out_data_006.tar

Run a search query and retrieve search results

If only some but not all files of one namespace or a set of files distributed over different namespaces shall be retrieved, it is reasonable to define a search query to find these files and retrieve them via their search id. Regular expression can be used to match multiple filenames but it cannot be used to match the path. Bash wildcards/globs do not work. In the example below, the files out_data_002.tar, out_data_005.tar and out_data_006.tar shall be retrieved:

$ slk list /ex/am/ple/data | cat
-rw-r--r--- k204221     bm0146          1.2M   10 Jun 2020  INDEX.txt
-rw-r--r--t k204221     bm0146         19.5G   05 Jun 2020  out_data_001.tar
-rw-r--r--t k204221     bm0146         19.0G   05 Jun 2020  out_data_002.tar
-rw-r--r--t k204221     bm0146         19.4G   05 Jun 2020  out_data_003.tar
-rw-r--r--t k204221     bm0146         19.3G   05 Jun 2020  out_data_004.tar
-rw-r--r--t k204221     bm0146         19.1G   05 Jun 2020  out_data_005.tar
-rw-r--r--t k204221     bm0146          7.8G   05 Jun 2020  out_data_006.tar
Files: 7

$ slk_helpers search_limited '{"$and": [{"path": {"$gte": "/ex/am/ple/data"}}, {"resources.name": {"$regex": "out_data_00[256].tar"}}]}'
slk list Search continuing. ..
Search ID: 65621

$ slk list 65621 | cat
-rw-r--r--t k204221     bm0146         19.0G   05 Jun 2020  out_data_002.tar
-rw-r--r--t k204221     bm0146         19.1G   05 Jun 2020  out_data_005.tar
-rw-r--r--t k204221     bm0146          7.8G   05 Jun 2020  out_data_006.tar
Files: 7

$ slk retrieve 65621 /tar/get/folder
[                                        ] 100% complete. Files retrieved: 3/3, [45.9G/45.9G].

$ ls /tar/get/folder
out_data_002.tar    out_data_005.tar    out_data_006.tar

In this example, the namespace /ex/am/ple/data and all of its sub-namespaces would be searched recursively. If the user wants the search to be performed non-recursively only in /ex/am/ple/data, then the operator $max_depth can be used.

$ slk_helpers search_limited '{"$and": [{"path": {"$gte": "/ex/am/ple/data", "$max_depth": 1}}, {"resources.name": {"$regex": "out_data_00[256].tar"}}]}'
slk list Search continuing. ..
Search ID: 65622

Instead of formulating the search query string yourself, you can use the command slk_helpers gen_file_query to generate such a query string.

# non-recursive search query string
$ slk_helpers gen_file_query /ex/am/ple/data/out_data_00[256].tar
'{"$and": [{"path": {"$gte": "/ex/am/ple/data", "$max_depth": 1}}, {"resources.name": {"$regex": "out_data_00[256].tar"}}]}'

# recursive search query string
$ slk_helpers gen_file_query -R /ex/am/ple/data/out_data_00[256].tar
'{"$and": [{"path": {"$gte": "/ex/am/ple/data"}}, {"resources.name": {"$regex": "out_data_00[256].tar"}}]}'

gen_file_query can only generate search query strings to search for resources (files/namespaces) by their name (field: resource.name) and/or location (path).

See also

More gen_file_query example applications are given :ref:here<gen_file_query_ex>.

Limits

slk retrieve SEARCH_ID TARGET_FOLDER will retrieve all files found by the search query SEARCH_ID into the same TARGET_FOLDER. Thus, if there are several files with same name but different source locations they will be written into TARGET_FOLDER a renamed as shown in the example below:

$ slk_helpers search_limited '{"$and": [{"path": {"$gte": "/ex/am/ple"}}, {"resources.name": "INDEX.txt"}]}'
slk list Search continuing. ..
Search ID: 65621

$ slk list 65621 | cat
-rw-r--r--- k204221     bm0146          1.6M   11 Sep 2020  INDEX.txt
-rw-r--r--- k204221     bm0146          1.2M   10 Jun 2020  INDEX.txt
-rw-r--r--- k204221     bm0146        924.9K   11 Sep 2020  INDEX.txt
-rw-r--r--- k204221     bm0146          1.3M   11 Sep 2020  INDEX.txt
-rw-r--r--- k204221     bm0146        924.9K   11 Sep 2020  INDEX.txt
Files: 5

$ slk retrieve 65621 .
[                                        ] 100% complete. Files retrieved: 5/5, [5.8M/5.8M].

$ ls
INDEX.DUPLICATE_FILENAME.49058705493.1.txt  INDEX.DUPLICATE_FILENAME.49058705494.1.txt  INDEX.DUPLICATE_FILENAME.49058705496.1.txt  INDEX.DUPLICATE_FILENAME.49058705520.1.txt  INDEX.txt

Striping

TLDR;

We recommend setting striping for your target directory if the files which you want to retrieve are 1 GB in size or larger. This will speed up the copying process from the HSM-Cache to the Lustre file system. Different striping settings are possible. The setting presented here is quite reasonable for applications at the DKRZ.

First, check if striping is already properly set. This is done with lfs getstripe -d PATH. In the example below, the folder example01 is not striped and the folder example02 is striped. If the folder is properly striped as example02 then nothing is to do.

$ lfs getstripe -d example01
stripe_count:  1 stripe_size:   1048576 pattern:       0 stripe_offset: -1

$ lfs getstripe -d example02
lcm_layout_gen:    0
lcm_mirror_count:  1
lcm_entry_count:   3
lcme_id:             N/A
lcme_mirror_id:      N/A
lcme_flags:          0
lcme_extent.e_start: 0
lcme_extent.e_end:   1073741824
    stripe_count:  1       stripe_size:   1048576       pattern:       raid0       stripe_offset: -1

lcme_id:             N/A
lcme_mirror_id:      N/A
lcme_flags:          0
lcme_extent.e_start: 1073741824
lcme_extent.e_end:   4294967296
    stripe_count:  4       stripe_size:   1048576       pattern:       raid0       stripe_offset: -1

lcme_id:             N/A
lcme_mirror_id:      N/A
lcme_flags:          0
lcme_extent.e_start: 4294967296
lcme_extent.e_end:   EOF
    stripe_count:  16       stripe_size:   1048576       pattern:       raid0       stripe_offset: -1

If the optiomal striping is not set, you can set it with lfs setstripe ... as follows:

$ mkdir example03
$ lfs setstripe -E 1G -c 1 -S 1M -E 4G -c 4 -S 1M -E -1 -c 8 -S 1M example03

Please check afterwards with lfs getstripe -d exmaple03 if the stripping setting looks like described above. If yes, all files which are written into the folder example03 will be automatically striped as defined. All files which existed in the folder before the lfs setstripe ... was applied will keep their old striping.

Details

On a Lustre file system, the content of files is served from so-called object storage targets (OSTs). Lustre allows configuring per file or per folder onto how many OSTs a file’s content is split. This configuration option is denoted as striping. striping=1 means that the file’s content is served by on OST only. striping=4 (for example) means that the file’s content is split into four distinct parts which are served by four OSTs – one part per OST. The general advantages and disadvantages of striping are not described here. There are numerous online sources on this topic.

The default configuration on the Lustre file system of Levante has still to be decided. For the time being, we suggest users to use a striping setting denoted as progressive file layout. This means that the striping factor is selected automatically depending on the file size. The proposal is to set striping to 1 for files smaller or equal 1 GB, to set striping to 4 for files between 1 GB and 4 GB in size and to set striping to 16 for files larger than 4 GB. The command to this looks as follows:

$ lfs setstripe -E 1G -c 1 -S 1M -E 4G -c 4 -S 1M -E -1 -c 8 -S 1M TARGET_FOLDER

Retrieval script templates

Three script templates for three use cases are printed below and available for download:

• several retrievals of single files: retrieve_slurm_template_single_files.sh

• one recursive retrieval of a whole namespace: retrieve_slurm_template_recursive.sh

• search and retrieval of search results: retrieve_slurm_template_search.sh

• auto-generate a search query, run it and retrieve search results: retrieve_slurm_template_gen_search.sh

When you use these templates, you need to make a few adaptions:

• replace /work/xz1234/ex/am/ple by the actual target folder on the Lustre file system

• replace xz1234 in --account=xz1234 by your project account name

• replace /path/to/your/archived/ by the namespace path to your data on the HSM

• replace /arch/bm0146/k204221/.*.nc by what you want to search for (only retrieve_slurm_template_gen_search.sh); Note: .* is a regular expression (RegEx) and no wildcard/glob. The search works only with RegEx and not with wildcards.

several archivals of single files

#!/bin/bash
# HINT:
# * You can change the values right of the "=" as you wish.
# * The "%j" in the log file names means that the job id will be inserted
#SBATCH --job-name=test_slk_retr_job # Specify job name
#SBATCH --output=test_job.o%j # name for standard output log file
#SBATCH --error=test_job.e%j # name for standard error output log
#SBATCH --partition=shared # partition name
#SBATCH --ntasks=1 # max. number of tasks to be invoked
#SBATCH --time=08:00:00 # Set a limit on the total run time
#SBATCH --account=xz1234 # Charge resources on this project
#SBATCH --mem=6GB

# ~~~~~~~~~~~~ preparation ~~~~~~~~~~~~
module load slk
# set target folder for retrieval
target_folder=/work/xz1234/ex/am/ple
# create folder to retrieve into (target folder)
mkdir -p ${target_folder}
# set striping for target folder
# see https://docs.dkrz.de/doc/hsm/striping.html
# ON LEVANTE
lfs setstripe -E 1G -c 1 -S 1M -E 4G -c 4 -S 1M -E -1 -c 8 -S 1M ${target_folder}
# ON MISTRAL
#lfs setstripe -S 4M -c 8 ${target_folder}

# ~~~~~~~~~~~~ retrievals ~~~~~~~~~~~~
# do the retrieval
echo "doing 'slk retrieve'"
# ~~~~~~~~~~~~ doing single-file retrievals ~~~~~~~~~~~~
# You can do multiple retrievals in one script, but based on our
# experience, 10 to 15 separate retrievals are enough to cover the
# 08:00 hours maximum time for running the slurm jobs. Therefore, it
# is recommended to run your retrieval scripts as follows:
# first retrieval and capture exit code (get $? in line after slk command)
slk retrieve /path/to/your/archived/file1 ${target_folder}
if [ $? -ne 0 ]; then
    >&2 echo "an error occurred in slk retrieve call"
else
    echo "retrieval successful"
fi

# second retrieval and capture exit code (get $? in line after slk cmd)
slk retrieve /path/to/your/archived/file2 ${target_folder}
if [ $? -ne 0 ]; then
    >&2 echo "an error occurred in slk retrieve call"
else
    echo "retrieval successful"
fi
# ...
# ...
# fifteenth retrieval and capture exit code (get $? in line after slk cmd)
slk retrieve /path/to/your/archived/file15 ${target_folder}
if [ $? -ne 0 ]; then
    >&2 echo "an error occurred in slk retrieve call"
else
    echo "retrieval successful"
fi

one recursive retrieval of a whole namespace

#!/bin/bash
# HINT:
# * You can change the values right of the "=" as you wish.
# * The "%j" in the log file names means that the job id will be inserted
#SBATCH --job-name=test_slk_retr_job # Specify job name
#SBATCH --output=test_job.o%j # name for standard output log file
#SBATCH --error=test_job.e%j # name for standard error output log
#SBATCH --partition=shared # partition name
#SBATCH --ntasks=1 # max. number of tasks to be invoked
#SBATCH --time=08:00:00 # Set a limit on the total run time
#SBATCH --account=xz1234 # Charge resources on this project
#SBATCH --mem=6GB

# ~~~~~~~~~~~~ preparation ~~~~~~~~~~~~
module load slk
# set target folder for retrieval
target_folder=/work/xz1234/ex/am/ple
# create folder to retrieve into (target folder)
mkdir -p ${target_folder}
# set striping for target folder
# see https://docs.dkrz.de/doc/hsm/striping.html
# ON LEVANTE
lfs setstripe -E 1G -c 1 -S 1M -E 4G -c 4 -S 1M -E -1 -c 8 -S 1M ${target_folder}
# ON MISTRAL
#lfs setstripe -S 4M -c 8 ${target_folder}

# ~~~~~~~~~~~~ doing recursive retrievals ~~~~~~~~~~~~
# If you wish to retrieve all files from a specific folder, you can use
# the recursive option (-R). Please use this option only of you need more
# than 90% of the files in a certain namespace directory, and don't
# retrieve a whole directory of e.g. 500 files for only 20 files. an
# example for recursive retrieval is as follows:
slk retrieve -R /path/to/your/archived/directory ${target_folder}
# '$?' captures the exit code of the previous command (you can put it in
# the next line after each slk command).
if [ $? -ne 0 ]; then
    >&2 echo "an error occurred in slk retrieve call"
else
    echo "retrieval successful"
fi

search and retrieval of search results

#!/bin/bash
# HINT:
# * You can change the values right of the "=" as you wish.
# * The "%j" in the log file names means that the job id will be inserted
#SBATCH --job-name=test_slk_retr_job # Specify job name
#SBATCH --output=test_job.o%j # name for standard output log file
#SBATCH --error=test_job.e%j # name for standard error output log
#SBATCH --partition=shared # partition name
#SBATCH --ntasks=1 # max. number of tasks to be invoked
#SBATCH --time=08:00:00 # Set a limit on the total run time
#SBATCH --account=xz1234 # Charge resources on this project
#SBATCH --mem=6GB

# ~~~~~~~~~~~~ preparation ~~~~~~~~~~~~
module load slk
# set target folder for retrieval
target_folder=/work/xz1234/ex/am/ple
# create folder to retrieve into (target folder)
mkdir -p ${target_folder}
# set striping for target folder
# see https://docs.dkrz.de/doc/hsm/striping.html
# ON LEVANTE
lfs setstripe -E 1G -c 1 -S 1M -E 4G -c 4 -S 1M -E -1 -c 8 -S 1M ${target_folder}
# ON MISTRAL
#lfs setstripe -S 4M -c 8 ${target_folder}

# ~~~~~~~~~~~~ doing the search ~~~~~~~~~~~~
# Set up a search that finds the files which you want to retrieve and
# and capture the resulting search id. See this FAQ entry for alternatives
# to capture the search id: "Can the search ID of slk search be captured
# by a shell variable?". We do this in two steps in order to be able to
# capture the exit code of the search command.
search_id_raw=`slk_helpers search_limited '{"$and": [{"path": {"$gte": "/path/to/your/archived"}}, {"resources.name": {"$regex": "out_data_00[256].tar"}}]}'`
# '$?' captures the exit code of the previous command (you can put it in
# the next line after each slk command).
if [ $? -ne 0 ]; then
    >&2 echo "an error occurred in slk search call"
else
    echo "search successful"
fi
search_id=`echo $search_id_raw | tail -n 1 | sed 's/[^0-9]*//g'`
echo "The search ID is ${search_id}"
#
# If we delimit the search query by `"` instead of `'` and escape all
# `$` and `"` in the query, then we might even use environment variables
# in the search query
#
#       source_namespace=/path/to/your/archived
#       slk_helpers search_limited "{\"\$and\": [{\"path\": {\"\$gte\": \"${source_namespace}\"}}, {\"resources.name\": {\"\$regex\": \"out_data_00[256].tar\"}}]}"
#

# ~~~~~~~~~~~~ doing recursive retrievals ~~~~~~~~~~~~
# If you wish to retrieve a set of files that have been found by a search
# you can use "slk retrieve SEARCH_ID TARGET_FOLDER"
slk retrieve ${search_id} ${target_folder}
# '$?' captures the exit code of the previous command (you can put it in
# the next line after each slk command).
if [ $? -ne 0 ]; then
    >&2 echo "an error occurred in slk retrieve call"
else
    echo "retrieval successful"
fi

generate search string and retrieve files

Download: retrieve_slurm_template_gen_search.sh

#!/bin/bash
# HINT:
# * You can change the values right of the "=" as you wish.
# * The "%j" in the log file names means that the job id will be inserted
#SBATCH --job-name=test_slk_retr_job # Specify job name
#SBATCH --output=test_job.o%j # name for standard output log file
#SBATCH --error=test_job.e%j # name for standard error output log
#SBATCH --partition=shared # partition name
#SBATCH --ntasks=1 # max. number of tasks to be invoked
#SBATCH --time=08:00:00 # Set a limit on the total run time
#SBATCH --account=xz1234 # Charge resources on this project
#SBATCH --mem=6GB

# ~~~~~~~~~~~~ preparation ~~~~~~~~~~~~
module load slk
# set target folder for retrieval
target_folder=/work/xz1234/ex/am/ple
# create folder to retrieve into (target folder)
mkdir -p ${target_folder}
# set striping for target folder
# see https://docs.dkrz.de/doc/hsm/striping.html
# ON LEVANTE
lfs setstripe -E 1G -c 1 -S 1M -E 4G -c 4 -S 1M -E -1 -c 8 -S 1M ${target_folder}
# ON MISTRAL
#lfs setstripe -S 4M -c 8 ${target_folder}

# ~~~~~~~~~~~~ doing the search ~~~~~~~~~~~~
search_query=`slk_helpers gen_file_query '/arch/bm0146/k204221/.*.nc'`
echo "The search query is ${search_query}"
search_id=$(eval "slk_helpers search_limited '"${search_query}"' | tail -n 1 | cut -c12-20")
echo "The search ID is ${search_id}"
#
# If we delimit the search query by `"` instead of `'` and escape all
# `$` and `"` in the query, then we might even use environment variables
# in the search query
#
#       source_namespace=/path/to/your/archived
#       slk_helpers search_limited "{\"\$and\": [{\"path\": {\"\$gte\": \"${source_namespace}\"}}, {\"resources.name\": {\"\$regex\": \"out_data_00[256].tar\"}}]}"
#

# ~~~~~~~~~~~~ doing recursive retrievals ~~~~~~~~~~~~
# If you wish to retrieve a set of files that have been found by a search
# you can use "slk retrieve SEARCH_ID TARGET_FOLDER"
slk retrieve ${search_id} ${target_folder}
# '$?' captures the exit code of the previous command (you can put it in
# the next line after each slk command).
if [ $? -ne 0 ]; then
    >&2 echo "an error occurred in slk retrieve call"
else
    echo "retrieval successful"
fi