Skip to main content
Ctrl+K
DKRZ Documentation  documentation - Home
  • Documentation
  • Blog
  • News
  • Home Page
  • Documentation
  • Blog
  • News
  • Home Page
Ctrl+K

Section Navigation

  • Getting started at DKRZ
    • Getting a user account
      • DKRZ user account
      • CERA user account
      • ESGF user account
      • Shared user account
    • Resource allocation
    • User Support
      • Help Desk
      • Data Support - Data Management
      • Training
        • Program Analysis and Tools Workshop
        • Training on DKRZ’s HPC systems (mistral, HPSS and cloud)
        • Introduction to DKRZ’s HLRE-3 system Mistral
    • Multi-Factor-Authentication
    • Terms of use
  • Levante HPC system
    • Configuration
    • Access and Environment
    • File Systems
    • Developing and Using Scientific Software
      • Compiling and Linking
      • Debugging
      • Building your own package: spack
      • Python
      • GPU Programming
    • Running Jobs with Slurm
      • Slurm Introduction
      • Partitions and Limits
      • Example Batch Scripts
      • Runtime Settings
      • Using GPU nodes
      • Accounting and Priorities
      • Slurm Binding Website
    • Data Processing on Levante
    • Data Transfer
      • uftp (Unicore FTP)
    • Remote file system
    • Known issues
    • Containers
      • Singularity
  • Data Storage
    • HPC System Storage
    • DKRZ HSM (tape archive)
      • Getting Started with slk
      • Known Issues (read this!)
      • Archivals to tape
      • Retrievals from tape
      • File Search and Metadata
      • slk usage examples
      • manual pages HSM
        • slk: official StrongLink Command Line Interface
        • slk helpers: slk extension provided by DKRZ
        • improved retrieval workflow v01
        • slk wrappers: SLURM wrapper scripts to simplify StrongLink-tasks
        • Reference: metadata schemata
        • Reference: StrongLink query language
        • JSON structure for/of metadata import/export
        • Reference: StrongLink verify jobs
        • Official StrongLink Command Line Interface Guide
      • FAQ
    • Swift Object Storage
      • Swiftbrowser
      • Python-swiftclient
    • S3 Object Storage
      • Storage Access
  • Software & Services
    • Jupyterhub
      • Overview
      • Quick Start
      • Spawner options
      • VNC Desktop
      • Kernels
      • Lab Extensions
      • Changelog
    • Gitlab
      • Gitlab-runner
    • Redmine
    • Data Science Services
      • Machine Learning on Levante
    • ModVis
    • ClusterCockpit
  • Data Services
    • Data Management Planning
    • Finding & Accessing Data
      • /pool/data user projects
      • ECMWF Reanalysis Products
      • Freva instances available at DKRZ
      • Freva
      • Do you need Data?
    • Processing & Analysing
    • Publishing & Dissemination
      • Data Preparation and Quality Checking
      • ESGF publication
      • Publication at WDCC
    • Archiving & Preserving
  • Visualization
    • ParaView
      • ParaView on Levante
      • Paraview Examples
        • Create an image of sea surface speeds
        • Display clouds as a semi-transparent overlay
        • Create isosurfaces of ocean current speed
        • Create a multi-variable image / animation from a coupled ice sheet - climate simulation
        • Prepare a regional land and ocean map background
        • Streamline seeding in vector data
        • Volume rendering of (NARVAL II) ICON data
        • Isosurfaces and isocontours in regular lat-lon data
      • DKRZ ParaView tutorial document
      • The Paraview main screen
      • Camera and perspective
        • Set the background (color/gradient)
        • Camera: Follow Path
        • Camera: Orbit
        • Switch to camera parallel projection
        • Camera: Interpolate camera locations
        • Set the view size
      • Colormaps
        • Hack the color map:
        • Chose a different colormap
        • Invert the Colormap
        • Adjust the color bar and its legend
        • Rescale the colormap to a custom range
        • Save the colormap as a preset
        • Adjust the color map for a transparent display of clouds
      • Export
        • Export an animation
        • Saving animations and screenshots
      • Filters
        • Combine files with AppendAttributes
        • Extrude a land surface based on topography
        • Apply a calculator to convert u and v into speed
        • Check if you have cell or point data
        • Use the contour filter to display isolines
        • Using an earth texture as background
        • Extrusion of topography and bathymetry
        • Compute isosurfaces from the point data
        • Add an image as texture (plane) in the background
        • Add an image as texture on a sphere
        • Use a threshold to get rid of missing missing values in ICON Ocean
        • NetCDF: Date and time annotation
      • Light
        • Deactivate the light kit
      • Readers
        • Load NEMO 3D data with the netCDF CF reader
        • The NetCDF CF reader
        • Load 2D ICON data with the CDI reader
        • Load 3D ICON Ocean data in Paraview
        • Pre-Processing for a multi-variable image / animation
      • Rendering Techniques
        • Use bump-mapping to composite current speed and SST
      • Technical
        • Adjust / check the camera controls
        • Open the Paraview settings
        • Activate the CDI reader plugin
        • Make Paraview save a state on quitting or crashing
        • Automating visualizations in ParaView with PvBatch
        • Save a state file
    • Python
    • NCL
      • NCL examples
    • PyNGL
    • GrADS
  • Documentation For Users
  • manual pages HSM
  • slk:...

slk: official StrongLink Command Line Interface#

file version: 05 Feb 2025

current software versions: slk version 3.3.91

The StrongLink Command Line Interface slk (Strong Link) consists of nine commands to interact with StrongLink and handle data stored in DKRZ’s hierarchal storage management system (HSM). slk does not provide an own shell like pftp but it is rather called like cdo on the Linux shell of you choice (cdo <OPERATOR> <=> slk <COMMAND> ...)

Download instructions for the official slk pdf manual: StrongLink Command Line Interface Guide.

Note

Namespace. StrongLink uses the term “namespace” or “global namespace” (gns). A “namespace” is comparable to a “directory” or “path” on a common file system.

slk help#

$ slk help

lists all commands

slk version#

$ slk version

print the current slk version

slk login#

$ slk login

Login with your LDAP credentials, i.e. DKRZ account number and password. Upon login, a personalized login token is created. This token is valid for 30 days during which time the user does not have to login again. This allows for using slk in non-interactive workflows, e.g. automated archival of model output. The expiration date of the token is stored in ~/.slk/config.json.

Note

slk stores the login token in the home directory of each user (~/.slk/config.json). By default, this file can only accessed by the respective user (permissions: -rw-------/600). However, users should be careful when doing things like chmod 755 * in their home directory. If you assume that your slk login token has been compromised please contact support@dkrz.de .

The functionality of slk login can be compared to Kerberos authentication in the HPSS.

The production StrongLink instance is set as default instance for login via slk login. If you need to login to another instance (you will be told by DKRZ staff if that is the case), you need to set the environment variables SLK_HOSTNAME and SLK_DOMAIN to values provided by the DKRZ staff.

slk archive#

$ slk archive [-p] [-R] [-v] [-vv] [-x] [--partsize S] [--stats] [--streams N] source_folder destination_gns
  • -p: preserve original file permission

  • -R: do recursive archival

  • -x: exclude files and folders which names start with a . (like .config or .git)

  • --partsize: size S of each file to upload per stream, default: 500

  • --stats: Display part & I/O stats for each node

  • --streams: number N of file part streams to use per node, default: 4 (please keep it at 4 and below)

  • -v: print simple progress bar of the archival status

  • -vv: print the name of each file that was processed and append status (SUCCESSFUL, SKIPPED or FAILED)

This command transfers files and directories from the Linux device to the HSM system. Appending the option -R performs a recursive archival as cp -r would do. A progress indicator is shown until the transfer is complete.

When -x (for exclude) is set all files and folders which names start with . are ignored. All files in such folders are also ignored. This helps omitting folders of your version control software or config files of your system.

The header of netCDF files is harvested during archival and written into the metadata database (see Metadata in StrongLink).

If a file is specified to be archived and a file with the same name already exists in the destination namespace, the size and timestamp of both files are compared. If both values are equal, the source file is not archived a second time. If one of them differs, the source file is archived and the old destination file is overwritten.

Note

Please do not run slk archive in regular intervals as sync tool - e.g. for daily synchronizations between your work folder and the tape archive. If your files on the Lustre file system are corrupted then your intact files on tape are overwritten by the corrupted ones.

slk list#

$ slk list [-a] [-b] [-n] [-R] [--text] GNS_PATH
$ slk list [-b] [-n] [--text] search_id
  • -a: use the -a to show “.” files

  • -b: show file sizes in byte

  • -n: print user id and group id instead of user name and group name

  • -R: do recursive listing (like ls -R)

  • --text: use the –text flag to print result to file (slk_${USER}_list.txt)

This command will display a listing of files and namespaces in the global namespace GNS_PATH or files corresponding to a completed search. The file name, size, owner, group, permissions and modification date and time will be displayed similar as ls -la does. A recursive listing is performed if -R is provided. The command slk list search_id will print all files found by the search with search id search_id. No path of the found files is printed.

A status indicator attached to the permissions string indicates whether a file is available in the cache or only on tape. It can be - (file in cache) or t (file not in cache but only in tape), e.g. -rwxr-xr-xt.

  • -: the file is located in the disk cache (and is therefore accessible in real-time)

  • t: the file is located on tape and has to be retrieved and copied to the disk cache prior to access

Note

slk search finds all files and namespaces that match the search query. slk list search_id prints only files (no namespaces) that the user is allowed to see/read. In contrast, slk list GNS_PATH lists files and directories in a namespace (GNS = namespace). Therefore, slk search and slk list search_id might report a different number of results for the same search.

slk search#

activated again since mid Oct. 2022

$ slk search "RQL Search Query"
$ slk search 'RQL Search Query'
$ slk search [-name FILE_NAME] [-user USER_NAME] [-group GROUP_NAME] [-partial]

This command will conduct a background search for files that match the specific query specified using a query language syntax that was designed by StrongLink. The query language is described on the Reference: StrongLink query language page and in the StrongLink Command Line Interface Guide from page 6 onwards.

Note

Operators in queries start with a $. If a query is delimited by " then the $ has to be escaped by a leading \ (\$OPERATOR). Otherwise, the operator is interpreted as environment variable by the shell. Alternatively, use ' as delimiter.

slk search offers the flags -name, -user and -group if a user just want to search for a specific file name, for files of a specific owner or for files belonging a specific group, respectively. These three flags can be combined with each other. These flags and a RQL search query are mutually exclusive – thus, only one of both can be used in one call of slk search.

A search result ID (search_id) will be returned. This ID can be used to list and retrieve files from the archive (see below).

One might need the user or group ids of respective users/groups to search files belonging to them. These ids are obtained as follows.

Get user id:

# get your user id
$ id -u

# get the id of any user
$ id USER_NAME -u

# get the id of any user
$ getent passwd USER_NAME
#  OR
$ getent passwd USER_NAME | awk -F: '{ print $3 }'

# get user name from user id
$ getent passwd USER_ID | awk -F: '{ print $1 }'

Get group id:

# get group ID from group name
$ getent group GROUP_NAME
#  OR
$ getent group GROUP_NAME | awk -F: '{ print $3 }'

# get group name from group id
$ getent group GROUP_ID | awk -F: '{ print $1 }'

# get groups and their ids of all groups of which member you are
$ id

Please see our documentation of specific Usage Examples (slk Usage Examples) and the StrongLink Command Line Interface Guide for exemplary calls of slk search.

slk retrieve#

Warning

This command is not recommended anymore. Please use slk_helpers retrieve or our retrieve watcher script instead.

Transfer a file from the HSM / tape archive to the user. When used on a login node, slk retrieve only allows to retrieve on file at once. Please see here for details.

slk retrieve [-d] [-ns] [-R] [-s] [--partsize S] [--stats] [--streams N] GNS_PATH destination_location
slk retrieve [-d] [-ns] [-R] [-s] [-R] [--partsize S] [--stats] [--streams N] GNS_PATH/ destination_location
slk retrieve [-d] [-ns] [-R] [-s] [--partsize S] [--stats] [--streams N] search_id destination_location
  • -d: create a duplicate if file exists, Default: false

  • -ns: preserve namespace in destination => whole source path is recreated in the target location, Default: false (true if used with a search_id)

  • -R: do recursive retrieval

  • -s: skip if file exists, Default: false

  • --partsize: size S of each file to upload per stream, default: 500

  • --stats: Display part & I/O stats for each node

  • --streams: number N of file part streams to use per node, default: 4 (please keep it at 4 and below)

This command will retrieve the files from the archive and copy them into the provided destination directory. A progress indicator will show until completed. The source files can be specified as a GNS Path or as a search result (via the search id). If recursive retrieval of all files into a namespace and its sub-namespaces is desired, please do not use slk retrieve but our new retrieve watcher script.

If a file should be retrieved and a file with the same name already exists in the target location, then the file is retrieved and the old file is overwritten. Please set -d to create a duplicate file locally (instead of overwriting) or set -s to skip retrieving an existing file.

slk delete#

$ slk delete [-R] GNS_PATH[,GNS_PATH[...]]
  • -R: do recursive deletion

The command slk delete works like rm in the Linux terminal. This command will delete GNS_PATH if it is a filre. If GNS_PATH is a namespace and -R is set, this command will delete GNS_PATH and its content. An error is thrown if GNS_PATH is a non-empty namespace and -R is not set.

slk tag#

$ slk tag [-R] GNS_PATH key=value
$ slk tag search_id key=value
$ slk tag [-display] [-R] GNS_PATH
$ slk tag [-display] search_id
  • -display: print metadata of files

  • -R: do recursive tagging

This command will add metadata to GNS_PATH (if this is a file), to all files in a GNS_PATH (if this is a directory) or to all files from a recent search. If -R is set, metadata of files in sub-namespaces will be also be updated. key has to be of the format metadata_schema_name.field_name. The available metadata schemata are listed on the page Reference: metadata schemata. Multiple key-value pairs can be appended.

slk group#

$ slk group [-R] GROUP_ID GNS_PATH
$ slk group [-R] GROUP_NAME GNS_PATH
  • -R: do recursive change of group

Change the group of a single file or a directory. If -R is provided, the group of the folder’s content will be changed recursively.

slk group can only be used by the file’s owner or an admin. Users can only set groups in which they are members. The Linux terminal command chgrp behaves the same.

To obtain the id of a group one can use one of these commands:

# get group ID and group members
$ getent group GROUP_NAME

# get groups and their ids of all groups of which you are a member
$ id

Note

slk group does not print a visible error message when it stops while modifying several files. Hence, it is important either to capture the exit code of slk group and slk owner or to have a look into the slk log (~/.slk/slk-cli.log) afterwards.

slk chmod#

$ slk chmod [-R] MODE GNS_PATH[,GNS_PATH[...]]
  • -R: do recursive change of permissions

Similar to chmod on the Linux terminal. MODE equals the MODE of bash chmod. Please see man chmod for details. If -R is provided, the permissions for the folder’s content are changed recursively.

slk move#

$ slk move [-i] SOURCE_GNS_PATH TARGET_GNS_PATH
  • -i: ask used whether a file should be overwritten (if in batch mode: skip file and throw an error)

Move a file or namespace defined by SOURCE_GNS_PATH into TARGET_GNS_PATH. Target files are overwritten without notice (new in slk version >3.3.21). Set -i to prevent automatic overwriting.

Note

Only a move into TARGET_GNS_PATH path is possible. No rename is possible with slk move in contrast to Unix/Linux mv (mv  old_path/old_file_name  new_path/new_file_name). Please us slk rename for renaming.

slk rename#

$ slk rename SOURCE_GNS_PATH TARGET_NAME

Rename a file or namespace defined by SOURCE_GNS_PATH to TARGET_NAME. TARGET_NAME has to be the new name and must not include the path (not slk rename path/old_file_name path/new_file_name, but slk rename path/old_file_name new_file_name).

slk recall#

Warning

This command is not recommended anymore. Please use slk_helpers recall or our recall watcher script instead.

Transfer a file from tape to the HSM cache.

slk recall [-R] GNS_PATH
slk recall search_id
  • -R: do recursive recall

This command will recall the files from tape into the HSM cache. The source files can be specified as a GNS Path or as a recent search result. If recursive recall of all files in a namespace and its sub-namespaces is desired, please use the option -R.

login token and further session information#

The login token and further session information are stored in ~/.slk/config.json. Please be aware that nobody except of you should have read or write permissions for this file. The permissions should be -rw-------.

If you want to use values of this file in your bash scripts, please consider using jq. Example: to obtain the expiration date of your token do jq .expireDate.

log file#

A central log file for all slk calls is located in ~/.slk/slk-cli.log.

environment variables#

# manually overwrite the default hostname (insert alternative hostname or IP address)
export SLK_HOSTNAME=HOSTNAME

# manually overwrite the default domain (default: "ldap")
export SLK_DOMAIN=ldap

Exit codes#

The lines with x in column contain unexpected exit codes.

command

task

exit code

attention

bad input command

always (redirected to slk help)

130

help

always

0

login

login successful

0

login not successful

1

search

search finished and search ID printed

0

path does not exist

1

RQL invalid

1

any error

1

list

content of namespace listed

0

namespace does not exist

1

search id does not exist

1

any error

1

retrieve

file(s) successfully transferred

0

target file did already exist and DUPLICATE was created

0

target file did already exist and was overwritten

0

at least one file was skipped

0

at least one file not successfully transferred

1

retrieval of more than 500 files requested

1

at least one source file has 0 byte size

slk hangs

x

any error

1

recall

file(s) successfully copied from tape to HSM cache

0

no file to recall (e.g. all files in HSM cache)

1

x

any error

1

archive

file(s) successfully transferred

0

at least one file was skipped

0

at least one file not successfully transferred

1

at least one source file has 0 byte size

0

any error

1

owner

ownership of file(s) changed successfully

0

at least one file was skipped

0

ownership of at least one file not changed successfully

1

any error

1

group

group of file(s) changed successfully

0

at least one file was skipped

0

group of at least one file not changed successfully

1

any error

1

chmod

permissions of file(s) changed successfully

0

at least one file was skipped

0

permissions of at least one file not changed successfully

1

any error

1

rename

resource (file/namespace) successfully renamed

0

source does not exist

1

target name does already exit

1

any error

1

move

resource (file/namespace) successfully moved

0

source does not exist

1

“user interaction: resource with same name exists in target location, overwrite? => yes”

0

“user interaction: resource with same name exists in target location, overwrite? => no”

0

any error

1

delete

resource (file(s)/namespace(s)) successfully deleted

0

target resource does not exist

1

any error

1

version

version printed successful

0

Known issues#

Please see the extra page Known Issues

On this page
  • slk help
  • slk version
  • slk login
  • slk archive
  • slk list
  • slk search
  • slk retrieve
  • slk delete
  • slk tag
  • slk group
  • slk chmod
  • slk move
  • slk rename
  • slk recall
  • login token and further session information
  • log file
  • environment variables
  • Exit codes
  • Known issues
Imprint and Privacy Policy

© Copyright Deutsches Klimarechenzentrum GmbH.

Created using Sphinx 7.4.7.

Built with the PyData Sphinx Theme 0.15.4.