slk: official StrongLink Command Line Interface#
file version: 31 Jan 2023
current software versions: slk version 3.3.83
The StrongLink Command Line Interface slk (Strong Link) consists of nine commands to interact with StrongLink and handle data stored in DKRZ’s hierarchal storage management system (HSM). slk does not provide an own shell like pftp but it is rather called like cdo on the Linux shell of you choice (cdo OPERATOR <=> slk <main class> ...)
Download instructions for the official slk pdf manual: StrongLink Command Line Interface Guide.
Note
Namespace. StrongLink uses the term “namespace” or “global namespace” (gns). A “namespace” is comparable to a “directory” or “path” on a common file system.
Note
Recursivity. The slk has a different interpretation of non-recursive commands compared to POSIX. If a directory/namespace is given as input to slk archive, slk retrieve, slk delete and slk tag, then all files in this folder are considered for archival/retrieval/deletion/tagging. Thus, slk ... SOURCE_DIR ... is comparable to POSIX compliant COMMAND SOURCE_DIR/* ... (COMMAND equals cp or rm). Full recursion is activated with the -R flag. The commands slk chmod, slk group and slk owner behave like chmod, chgrp and chown, respectively.
slk help#
$ slk help
lists all commands
slk version#
$ slk version
print the current slk version
slk login#
$ slk login
Login with your LDAP credentials, i.e. DKRZ account number and password. Upon login, a personalized login token is created. This token is valid for 30 days during which time the user does not have to login again. This allows for using slk in non-interactive workflows, e.g. automated archival of model output. The expiration date of the token is stored in ~/.slk/config.json.
Note
slk stores the login token in the home directory of each user (~/.slk/config.json). By default, this file can only accessed by the respective user (permissions: -rw-------/600). However, users should be careful when doing things like chmod 755 * in their home directory. If you assume that your slk login token has been compromised please contact support@dkrz.de .
The functionality of slk login can be compared to Kerberos authentication in the HPSS.
The production StrongLink instance is set as default instance for login via slk login. If you need to login to another instance (you will be told by DKRZ staff if that is the case), you need to set the environment variables SLK_HOSTNAME and SLK_DOMAIN to values provided by the DKRZ staff.
slk archive#
$ slk archive [-p] [-R] [-v] [-vv] [-x] [--partsize S] [--stats] [--streams N] source_folder destination_gns
-p: preserve original file permission-R: do recursive archival-x: exclude files and folders which names start with a.(like.configor.git)--partsize: sizeSof each file to upload per stream, default: 500--stats: Display part & I/O stats for each node--streams: numberNof file part streams to use per node, default: 4 (please keep it at 4 and below)-v: print simple progress bar of the archival status-vv: print the name of each file that was processed and append status (SUCCESSFUL,SKIPPEDorFAILED)
This command transfers files and directories from the Linux device to the HSM system. Appending the option -R performs a recursive archival as cp -r would do. A progress indicator is shown until the transfer is complete.
When -x (for exclude) is set all files and folders which names start with . are ignored. All files in such folders are also ignored. This helps omitting folders of your version control software or config files of your system.
The header of netCDF files is harvested during archival and written into the metadata database (see Metadata in StrongLink).
If a file is specified to be archived and a file with the same name already exists in the destination namespace, the size and timestamp of both files are compared. If both values are equal, the source file is not archived a second time. If one of them differs, the source file is archived and the old destination file is overwritten.
Note
Please do not run slk archive in regular intervals as sync tool - e.g. for daily synchronizations between your work folder and the tape archive. If your files on the Lustre file system are corrupted then your intact files on tape are overwritten by the corrupted ones.
slk list#
$ slk list [-a] [-b] [-n] [-R] [--text] GNS_PATH
$ slk list [-b] [-n] [--text] search_id
-a: use the -a to show “.” files-b: show file sizes in byte-n: print user id and group id instead of user name and group name-R: do recursive listing (likels -R)--text: use the –text flag to print result to file (slk_${USER}_list.txt)
This command will display a listing of files and namespaces in the global namespace GNS_PATH or files corresponding to a completed search. The file name, size, owner, group, permissions and modification date and time will be displayed similar as ls -la does. A recursive listing is performed if -R is provided. The command slk list search_id will print all files found by the search with search id search_id. No path of the found files is printed.
A status indicator attached to the permissions string indicates whether a file is available in the cache or only on tape. It can be - (file in cache) or t (file not in cache but only in tape), e.g. -rwxr-xr-xt.
-: the file is located in the disk cache (and is therefore accessible in real-time)t: the file is located on tape and has to be retrieved and copied to the disk cache prior to access
Note
slk search finds all files and namespaces that match the search query. slk list search_id prints only files (no namespaces) that the user is allowed to see/read. In contrast, slk list GNS_PATH lists files and directories in a namespace (GNS = namespace). Therefore, slk search and slk list search_id might report a different number of results for the same search.
slk search#
activated again since mid Oct. 2022
$ slk search "RQL Search Query"
$ slk search 'RQL Search Query'
$ slk search [-name FILE_NAME] [-user USER_NAME] [-group GROUP_NAME] [-partial]
This command will conduct a background search for files that match the specific query specified using a query language syntax that was designed by StrongLink. The query language is described on the Reference: StrongLink query language page and in the StrongLink Command Line Interface Guide from page 6 onwards.
Note
Operators in queries start with a $. If a query is delimited by " then the $ has to be escaped by a leading \ (\$OPERATOR). Otherwise, the operator is interpreted as environment variable by the shell. Alternatively, use ' as delimiter.
slk search offers the flags -name, -user and -group if a user just want to search for a specific file name, for files of a specific owner or for files belonging a specific group, respectively. These three flags can be combined with each other. These flags and a RQL search query are mutually exclusive – thus, only one of both can be used in one call of slk search.
A search result ID (search_id) will be returned. This ID can be used to list and retrieve files from the archive (see below).
One might need the user or group ids of respective users/groups to search files belonging to them. These ids are obtained as follows.
Get user id:
# get your user id
$ id -u
# get the id of any user
$ id USER_NAME -u
# get the id of any user
$ getent passwd USER_NAME
# OR
$ getent passwd USER_NAME | awk -F: '{ print $3 }'
# get user name from user id
$ getent passwd USER_ID | awk -F: '{ print $1 }'
Get group id:
# get group ID from group name
$ getent group GROUP_NAME
# OR
$ getent group GROUP_NAME | awk -F: '{ print $3 }'
# get group name from group id
$ getent group GROUP_ID | awk -F: '{ print $1 }'
# get groups and their ids of all groups of which member you are
$ id
Please see our documentation of specific Usage Examples (slk Usage Examples) and the StrongLink Command Line Interface Guide for exemplary calls of slk search.
slk retrieve#
Note
When used on a login node, slk retrieve only allows to retrieve on file at once. Please see here for details.
slk retrieve [-d] [-ns] [-R] [-s] [--partsize S] [--stats] [--streams N] GNS_PATH destination_location
slk retrieve [-d] [-ns] [-R] [-s] [-R] [--partsize S] [--stats] [--streams N] GNS_PATH/ destination_location
slk retrieve [-d] [-ns] [-R] [-s] [--partsize S] [--stats] [--streams N] search_id destination_location
-d: create a duplicate if file exists, Default: false-ns: preserve namespace in destination => whole source path is recreated in the target location, Default: false (true if used with asearch_id)-R: do recursive retrieval-s: skip if file exists, Default: false--partsize: sizeSof each file to upload per stream, default: 500--stats: Display part & I/O stats for each node--streams: numberNof file part streams to use per node, default: 4 (please keep it at 4 and below)
This command will retrieve the files from the archive and copy them into the provided destination directory. A progress indicator will show until completed. The source files can be specified as a GNS Path or as a search result (via the search id). If recursive retrieval of all files into a namespace and its sub-namespaces is desired, please use the option -R. slk retrieve will throw an error and quit immediately, if more than 500 files are to be retrieved at once.
Note
We strongly recommend using striping on Levante for the time being. Some folders are striped already. Please check in advance.
Note
Currently (slk 3.3.81), exit code 1 is returned when one or more files are skipped. This will be changed to exit code 0 in future.
The notation of the source directory in slk retrieve is similar to rsync: slk retrieve GNS_PATH/ destination_location is different to slk retrieve GNS_PATH destination_location (no slash). GNS_PATH/ retrieves the content of GNS_PATH without creating a folder in destination_location that equals the most right folder in GNS_PATH. Instead, slk retrieve GNS_PATH destination_location` (no slash in the end) equals mkdir destination_location/`basename GNS_PATH` and slk retrieve GNS_PATH/ destination_location/`basename GNS_PATH`.
If a file should be retrieved and a file with the same name already exists in the target location, then the file is retrieved and the old file is overwritten. Please set -d to create a duplicate file locally (instead of overwriting) or set -s to skip retrieving an existing file.
slk delete#
$ slk delete [-R] GNS_PATH[,GNS_PATH[...]]
-R: do recursive deletion
The command slk delete works like rm in the Linux terminal. This command will delete GNS_PATH if it is a filre. If GNS_PATH is a namespace and -R is set, this command will delete GNS_PATH and its content. An error is thrown if GNS_PATH is a non-empty namespace and -R is not set.
slk tag#
$ slk tag [-R] GNS_PATH key=value
$ slk tag search_id key=value
$ slk tag [-display] [-R] GNS_PATH
$ slk tag [-display] search_id
-display: print metadata of files-R: do recursive tagging
This command will add metadata to GNS_PATH (if this is a file), to all files in a GNS_PATH (if this is a directory) or to all files from a recent search. If -R is set, metadata of files in sub-namespaces will be also be updated. key has to be of the format metadata_schema_name.field_name. The available metadata schemata are listed on the page Reference: metadata schemata. Multiple key-value pairs can be appended.
slk group#
$ slk group [-R] GROUP_ID GNS_PATH
$ slk group [-R] GROUP_NAME GNS_PATH
-R: do recursive change of group
Change the group of a single file or a directory. If -R is provided, the group of the folder’s content will be changed recursively.
slk group can only be used by the file’s owner or an admin. Users can only set groups in which they are members. The Linux terminal command chgrp behaves the same.
To obtain the id of a group one can use one of these commands:
# get group ID and group members
$ getent group GROUP_NAME
# get groups and their ids of all groups of which you are a member
$ id
Note
slk group does not print a visible error message when it stops while modifying several files. Hence, it is important either to capture the exit code of slk group and slk owner or to have a look into the slk log (~/.slk/slk-cli.log) afterwards.
slk chmod#
$ slk chmod [-R] MODE GNS_PATH[,GNS_PATH[...]]
-R: do recursive change of permissions
Similar to chmod on the Linux terminal. MODE equals the MODE of bash chmod. Please see man chmod for details. If -R is provided, the permissions for the folder’s content are changed recursively.
slk move#
$ slk move [-i] SOURCE_GNS_PATH TARGET_GNS_PATH
-i: ask used whether a file should be overwritten (if in batch mode: skip file and throw an error)
Move a file or namespace defined by SOURCE_GNS_PATH into TARGET_GNS_PATH. Target files are overwritten without notice (new in slk version >3.3.21). Set -i to prevent automatic overwriting.
Note
Only a move into TARGET_GNS_PATH path is possible. No rename is possible with slk move in contrast to Unix/Linux mv (mv old_path/old_file_name new_path/new_file_name). Please us slk rename for renaming.
slk rename#
$ slk rename SOURCE_GNS_PATH TARGET_NAME
Rename a file or namespace defined by SOURCE_GNS_PATH to TARGET_NAME. TARGET_NAME has to be the new name and must not include the path (not slk rename path/old_file_name path/new_file_name, but slk rename path/old_file_name new_file_name).
slk recall#
Only needed when external data access via archive.dkrz.de should take place (for more details read: External Access).
slk recall [-R] GNS_PATH
slk recall search_id
-R: do recursive recall
This command will recall the files from tape into the HSM cache. The source files can be specified as a GNS Path or as a recent search result. If recursive recall of all files in a namespace and its sub-namespaces is desired, please use the option -R.
login token and further session information#
The login token and further session information are stored in ~/.slk/config.json. Please be aware that nobody except of you should have read or write permissions for this file. The permissions should be -rw-------.
If you want to use values of this file in your bash scripts, please consider using jq. Example: to obtain the expiration date of your token do jq .expireDate.
log file#
A central log file for all slk calls is located in ~/.slk/slk-cli.log.
environment variables#
# manually overwrite the default hostname (insert alternative hostname or IP address)
export SLK_HOSTNAME=HOSTNAME
# manually overwrite the default domain (default: "ldap")
export SLK_DOMAIN=ldap
Exit codes#
The lines with x in column contain unexpected exit codes.
command |
task |
exit code |
attention |
|---|---|---|---|
bad input command |
always (redirected to slk help) |
130 |
|
help |
always |
0 |
|
login |
login successful |
0 |
|
login not successful |
1 |
||
search |
search finished and search ID printed |
0 |
|
path does not exist |
1 |
||
RQL invalid |
1 |
||
any error |
1 |
||
list |
content of namespace listed |
0 |
|
namespace does not exist |
1 |
||
search id does not exist |
1 |
||
any error |
1 |
||
retrieve |
file(s) successfully transferred |
0 |
|
target file did already exist and DUPLICATE was created |
0 |
||
target file did already exist and was overwritten |
0 |
||
at least one file was skipped |
0 |
||
at least one file not successfully transferred |
1 |
||
retrieval of more than 500 files requested |
1 |
||
at least one source file has 0 byte size |
slk hangs |
x |
|
any error |
1 |
||
recall |
file(s) successfully copied from tape to HSM cache |
0 |
|
no file to recall (e.g. all files in HSM cache) |
1 |
x |
|
any error |
1 |
||
archive |
file(s) successfully transferred |
0 |
|
at least one file was skipped |
0 |
||
at least one file not successfully transferred |
1 |
||
at least one source file has 0 byte size |
0 |
||
any error |
1 |
||
owner |
ownership of file(s) changed successfully |
0 |
|
at least one file was skipped |
0 |
||
ownership of at least one file not changed successfully |
1 |
||
any error |
1 |
||
group |
group of file(s) changed successfully |
0 |
|
at least one file was skipped |
0 |
||
group of at least one file not changed successfully |
1 |
||
any error |
1 |
||
chmod |
permissions of file(s) changed successfully |
0 |
|
at least one file was skipped |
0 |
||
permissions of at least one file not changed successfully |
1 |
||
any error |
1 |
||
rename |
resource (file/namespace) successfully renamed |
0 |
|
source does not exist |
1 |
||
target name does already exit |
1 |
||
any error |
1 |
||
move |
resource (file/namespace) successfully moved |
0 |
|
source does not exist |
1 |
||
“user interaction: resource with same name exists in target location, overwrite? => yes” |
0 |
||
“user interaction: resource with same name exists in target location, overwrite? => no” |
0 |
||
any error |
1 |
||
delete |
resource (file(s)/namespace(s)) successfully deleted |
0 |
|
target resource does not exist |
1 |
||
any error |
1 |
||
version |
version printed successful |
0 |
Known issues#
Please see the extra page Known Issues