Getting Started with slk¶
file version: 13 November 2021
current slk version: 3.3.16
slk_helpers comands behave very similar to known commands on the Linux terminal (POSIX compliant):
chown(only admins; currently deactivated)
slk archiveautomatically creates namespaces)
slk_helpers checksum -t sha512=>
test -e(also returns the resource id)
slk_helpers size=> get file size, e.g. equal to
However, there are a few differences which might cause issues if users expect a fully POSIX-compliant slk. Please read slk pitfalls for details. What we know as a
directory is called
namespace in StrongLink. Not all
slk_helpers commands are listed above. Please have a look into the slk manual and slk_helpers manual to get a list of all available commands.
Before you can use the slk commands the first time, you must call
slk login and login with your DKRZ/mistral/luv login credentials.
Load slk module¶
slk is available as module on all mistral nodes. Just do
$ module load slk
to load the most recent slk version. If you wish to use a specific old release (e.g. slk version 3.1.58), please do
$ module load slk/3.1.58
On which nodes to run slk¶
We suggest running
slk archive and
slk retrieve on the
compute2 nodes. The run time on the
mistralpp nodes considerably depends on the activity of other users on these nodes.
Please do not run
slk archive and
retrieve on the mistral login nodes (
mlogin10X) when you archive large amounts of data because
slk causes high CPU load and uses much memory.
The available memory per job on the
shared nodes is very low. Therefore,
slk archive and
slk retrieve are slower than on other nodes. The run time can be expected to be two to four times as long as on the
slk login and use your common DKRZ credentials to log in. You do not login to another shell with
slk login (like
pftp) but it creates a login token. This login token allows the usage of the other slk commands. It is located in
~/.slk/config.json and is valid for 30 days.
$ slk login
All mistral nodes share the same home directory. Therefore, you need to login only once and you can run it on any mistral node.
Archive a file or directory¶
slk archive can be used to archive files or directories. A progress bar is printed when
slk archive is used in interactive mode (session on
mistralpp or in an interactive job via
slk archive is used in a batch script, no output will be written into the job log. An example
slk archive call would be:
$ slk archive /work/bm0146/k204221/some_file.nc /arch/bm0146/k204221/my_data/
slk archive allows the usage of
* as wildcard (but not
-R to do recursive archival of directories.
slk archive works semi-recursive without
-R (see slk pitfalls for details).
List content of a namespace¶
slk list automatically prints its findings in a pagination mode with 25 items per page. That means that you see the first 25 results and have to type
Enter to show the next 25 results. The pagination mode is deactivated when the output of
slk list is piped (
|) into another command – e.g.
$ slk list /arch/bm0146/k204221/my_data | cat -rw-rw-rw-- k204221 bm0146 1.2K 27 Mar 2020 borehole_01.nc -rw-rw-rw-- k204221 bm0146 1.2K 04 Mar 2021 nc3.nc -rw-rw-rw-- k204221 bm0146 1.2K 04 Mar 2021 nc_k_2.nc -rw-rw-rw-- k204221 bm0146 4.0M 04 Mar 2021 nc_k_3.nc -rw-rw-rw-- k204221 bm0146 4.0M 04 Mar 2021 nc_k_4.nc -rw-rw-rw-- k204221 bm0146 13.1K 08 Dec 2020 small.nc -rwxrwx-w-- k204221 bm0146 105.5M 08 Nov 2019 small_BPb4-Sl-mT_00062104_00040000000_01040000000.AGM07807972.freeze.nc -rwxrwx-w-- k204221 bm0146 105.5M 14 Nov 2019 small_BPb4-Sl-mT_00062104_00040000000_01040000000.AGM07807972.nc -rw-rw-rw-- k204221 bm0146 1.2K 23 Mar 2021 test.nc -rw-rw-rw-- k204221 bm0146 1.2K 23 Mar 2021 zonk.nc -rw-rw-rw-- k204221 bm0146 1.2K 28 Jun 2021 some_file.nc Files: 11
Retrieve a file or namespace¶
slk retrieve is the counterpart of
slk archive. It behaves quite the same as
slk archive (progress bar;
-R) but the * works only for files and not for namesspaces.
slk retrieve /arch/bm0146/k204221/my_data/nc3.nc /work/bm0146/k204221/results
Move, rename and delete files or namespaces¶
Move a file from one namespace to another. The file’s name cannot be changed by this command.
$ slk move /arch/bm0146/k204221/my_data/nc_k_2.nc /arch/bm0146/k204221/old_data
Rename a file. The file’s location cannot be changed by this command.
$ slk rename /arch/bm0146/k204221/my_data/nc_k_3.nc a_netcdf_file.nc
Delete a file. If we apply
slk delete onto a namespace then it deletes all files in this namespace without confirmation. Files in sub-namespaces are not deleted. To delete the whole namespace with all sub-, subsub-, …-namespaces and their content, please append
$ slk delete /arch/bm0146/k204221/my_data/zonk.nc
Check the exit code of slk by reading out
$? which prints the exit code of the previously called command.
See the log file in
~/.slk/slk-cli.log for detailed error messages
Packems was adapted to slk and the new HSM system. Please have a look into the packems manual for details and usage of packems: https://code.mpimet.mpg.de/projects/esmenv/wiki/Packems.
slk_helpers commands are available as functions
pyslk.pyslk.slk_COMMAND(...). All command lines arguments of
slk version 3.3.6 and later are available as the functions’ arguments. These functions are simple wrappers that print out the text, which the
slk_helpers commands normally print to the command line. A bit more advanced wrappers are available for a few commands via
pyslk.parsers.slk_COMMAND_.... Feel free to download the package from pyslk downloads.
A few usage examples:
> from pyslk import pyslk as pslk > from pyslk import parsers as psr > pslk.slk_list('/arch') drwxrwxr-x- 7003 1001 06 Aug 2021 aa0049 drwxrwxr-x- 7003 1151 06 Aug 2021 aa0238 drwxrwx-w-- 7003 1079 06 Aug 2021 ab0036 drwxrwx-w-- 7003 1007 06 Aug 2021 ab0051 ... > pslk.slk_version() SCLI Version 3.3.10 > psr.slk_list_formatted('/arch') permissions owner group size day month year filename 0 drwxrwxr-x- a270003 aa0049 06 Aug 2021 aa0049 1 drwxrwxr-x- a270003 aa0238 06 Aug 2021 aa0238 2 drwxrwx-w-- a270003 ab0036 06 Aug 2021 ab0036 3 drwxrwx-w-- a270003 ab0051 06 Aug 2021 ab0051 4 ... >