slk helpers: slk extension provided by DKRZ

file version: 14 June 2022

current software versions: slk version 3.3.21; slk_helpers version 1.2.4

The slk_helpers is an extensions to the slk. The slk is developed by StrongBox Data Solutions and belongs to the StrongLink HSM software. The slk_helpers have been developed at the DKRZ to provide some useful functionality that is not included in the slk. If specific usage information is missing on this help page or if you encounter errors, please contact support@dkrz.de.

Note

StrongLink uses the term “namespace” or “global namespace” (gns). A “namespace” is comparable to a “directory” or “path” on a common file system.

slk_helpers help

$ slk_helpers help

lists all commands

slk version

$ slk version

print the current slk_helpers version

slk_helpers checksum

$ slk_helpers checksum [-t CHECKSUM_TYPE] RESOURCE_PATH
  • -t, --type: checksum_type (possible values: sha512, adler32); omit to print all available checksums

Prints the checksum(s) of a resource. If -t is set, the checksum of type CHECKSUM_TYPE is retrieved. Possible values are sha512 and adler32. If -t is not set, all available checksums are printed. It only works for files and not for namespaces. Namespaces do not have checksums.

StrongLink calculates two checksums of each archived file and stores them in the metadata. It compares the stored checksums with the file’s actual checksums at certain stages of the archival and retrieval process. Commonly, users do not need to check the checksum manually. But, you can if you prefer to do it. If a file has no checksum then it has not been fully archived yet (e.g. the copying is still in progress; archival process canceled).

slk_helpers exists

$ slk_helpers exists RESOURCE_PATH

Check if the resource RESOURCE_PATH exists. The resource id is returned if it exists. exists works for files and namespaces.

slk_helpers gen_file_query

$ slk_helpers gen_file_query [-R] RESOURCE1 [RESOURCE2 [RESOURCE3 [...]]]
  • -R, --recursive: generate a query which does a recursive search

Generates a search query which can be used with slk search and slk_helpers search_limited to perform a search for the resources RESOURCE1, RESOURCE2, … . These can be either files or namespaces. If a filename without path is provided, then the file will be searched for everywhere in the HSM. Filenames may contain regular expressions but no bash wildcards/globs. The path to a file must not contain regular expressions. Detailled examples and explanations are given in Generate search queries for filenames.

slk_helpers hostname

$ slk_helpers hostname

Prints the hostname to which slk is currently connected to or to which slk will connect. It should be archive.dkrz.de. This is the default value on each Levante node. You can overwrite the default hostname by exporting the environment variable SLK_HOSTNAME (e.g. by export SLK_HOSTNAME=stronglink.hsm.dkrz.de on bash).

slk_helpers iscached

$ slk_helpers iscached RESOURCE_PATH

Check if the resource RESOURCE_PATH is stored in the HSM cache. The user is informed via a text message whether RESOURCE_PATH exists. Additionally, the exit code will be 0 if the resource is in the cache and 1 if not (exit code: get the variable $? directly after the slk call). If a file is not stored in the cache then it is only stored on tape. Retrievals from tape will take considerable longer than retrievals from cache.

slk_helpers metadata

$ slk_helpers metadata RESOURCE_PATH

Prints the available metadata of a resource. Corresponds to slk tag – whereas slk tag sets metadata and slk_helpers metadata prints metadata.

slk_helpers mkdir

$ slk_helpers mkdir [-R] GNS_PATH
  • -R: use the -R create folders recursively, if the parent folders do not exist; Default: false

Create a namespace in an already existing namespace (== create basename GNS_PATH in dirname GNS_PATH). This command works like mkdir on a Linux terminal. Create nested namesspaces recursively when -R is set (like mkdir -r on Linux terminal).

slk_helpers resourcepath

$ slk_helpers resourcepath RESOURCE_ID

Get path for a resource id

slk_helpers search_limited

$ slk search "RQL Search Query"
$ slk search 'RQL Search Query'

This command will conduct a background search for files that match the specific query specified using a query language syntax that was designed by StrongBox Data Solutions. The query language is described on the StrongLink query language page and in the StrongLink Command Line Interface Guide from page 6 onwards.

Note

Operators in queries start with a $. If a query is delimited by " then the $ has to be escaped by a leading \ (\$OPERATOR). Otherwise, the operator is interpreted as environment variable by the shell. Alternatively, use ' as delimiter.

A search result ID (search_id) will be returned if 1000 or less results were found. If more results were found, an error and no search ID will be printed. 1000 refers to the total number of results of which some might not be visible to the user. The search ID can be used to list and retrieve files from the archive (see below).

One might need the user or group ids of respective users/groups to search files belonging to them. These ids are obtained as follows.

Get user id:

# get your user id
$ id -u

# get the id of any user
$ id USER_NAME -u

# get the id of any user
$ getent passwd USER_NAME
#  OR
$ getent passwd USER_NAME | awk -F: '{ print $3 }'

# get user name from user id
$ getent passwd USER_ID | awk -F: '{ print $1 }'

Get group id:

# get group ID from group name
$ getent group GROUP_NAME
#  OR
$ getent group GROUP_NAME | awk -F: '{ print $3 }'

# get group name from group id
$ getent group GROUP_ID | awk -F: '{ print $1 }'

# get groups and their ids of all groups of which member you are
$ id

Please see our documentation of specific Usage Examples (slk Usage Examples) and the StrongLink Command Line Interface Guide for examplary calls of slk search. These can be 1:1 used with slk_helpers search_limited.

Note

slk_helpers search_limited counts all files and namespaces that match the search query and that the current user is allowed to see/read. slk list lists only the respective files. Therefore, slk_helpers search_limited might print an error that more than 1000 resources were found although there are less than 1000 matches of which the user has read permissions. Moreover, different users might get different output of slk list for the same search id.

slk_helpers session

$ slk_helpers session

Prints until when the current slk session is valid.

slk_helpers size

$ slk_helpers size

Returns file size in byte

Exit codes

command

task

exit code

bad input command

always (redirected to slk help)

2

general

(not help, version and session)

session expired

2

issue related to config file

2

help

always

0

mkdir

namespace successfully created

0

namespace with same name already exists

1

any other error

2

exists

resource exists

0

resource does not exist

1

any error

2

session

login token exists and is not expired

0

no login token

1

session expired

1

hostname

hostname is set and is as printed

0

any error

2

checksum

resource exists and has checksum

0

resource not found

1

requested checksum not available

1

any other error

2

metadata

resource exists and metadata available

0

resource does not exist

1

any error

2

resourcepath

resource with given ID exists

0

resource with given ID does not exist

1

any error

2

size

resource exists and has size

0

resource does not exist

1

any error

2

version

always

0

iscached

resource exists and is cached

0

resource exists and is not cached

1

resource does not exist

2

any error

2

search_limited

search successfully performed

0

any error

2

list_search

search id correct and search results to print

0

search id correct but no results to print

1

any error

2

gen_file_query

query successfully generated

0

any error

2

Technical background of selected commands

slk_helpers gen_file_query

TODO

Major Changes

1.2.x

  • new commands
    • gen_file_query: create a query string to search files, which are provided as input

    • list_search: list search results (incl. path of resources)

    • updated exit codes

1.1.x

  • new commands
    • iscached: prints out whether a file is cached (== quick access) or not

    • search_limited: like slk search put works only for searches that 1000 results or less)

    • version: prints the version of slk_helpers