Switching: pftp to slk¶
file version: 28 February 2022
current slk version: 3.3.21
Some common pftp
workflows and their counterparts using slk
are listed below to simplify switching from pftp
to slk
. A general description of slk
is given in StrongLink Command Line Interface (slk) and additional example calls are provided in slk Usage Examples.
Nomenclature¶
We use some abbreviated file and folder names in the code examples for simpler readability. These are
/lustre_dir
: a directory path on the lustre file system (e.g/work/bm0146/k204221
)/tape_dir
: a directory path in the tape archive (e.g today/hpss/arch/bm0146/k204221
)file.nc
: a netCDF file
Login¶
pftp¶
$ pftp
...
ftp>
slk¶
Login to another shell like to the ftp
shell via pftp
is not necessary. Instead, the user has to generate a login token every 30 days. This procedure is comparable to the Kerberos authentication. The slk
commands are run on the normal Linux shell.
$ slk login
Username: XYZ
Password:
Login Successful
Note
slk
stores the login token in the home directory of each user (~/.slk/config.json
). By default, this file can only be accessed by the respective user (default permissions: -rw-------
/600
). However, users should be careful when doing things like chmod 755 *
in their home directory so that other users cannot access the token. If you assume that your slk login token has been compromized please contact support@dkrz.de .
Archival of a single file¶
pftp¶
$ cd /lustre_dir
$ pftp
# possibly `mkdir tape_dir`
ftp> cd /tape_dir
ftp> put file.nc
ftp> quit
slk¶
$ slk archive /lustre_dir/file.nc /tape_dir
Archival of a whole directory¶
pftp¶
$ cd /lustre_dir
$ pftp
# possibly `mkdir /tape_dir`
ftp> cd /tape_dir
ftp> prompt
ftp> mput *
ftp> quit
slk¶
$ slk archive /lustre_dir /tape_dir
All files located in /lustre_dir
are archived. No recursive archival of directories located in /lustre_dir
is performed. If you wish to do a recursive archival of the whole folder tree below /lustre_dir
, just add the -R
option as follows:
$ slk archive -R /lustre_dir /tape_dir
Note
You do not need to add *
to the source path as would be necessary for cp
(cp /lustre_dir/* /tape_dir/`basename /lustre_dir`
). If you add *
to the source path in the slk archive
call (slk archive /lustre_dir/* /tape_dir
) then you would archive all sub-directories with their content. The sub-directories/namespaces are automatically created in StrongLink. If you have sub-folders 1900
until 1999
(one output folder per model year) and you only want to archive the 1980s then you simply do slk archive /lustre_dir/198? /tape_dir
.
Retrieval of a single file¶
pftp¶
$ cd /lustre_dir
$ pftp
ftp> cd /tape_dir
ftp> get file.nc
ftp> quit
slk¶
$ slk retrieve /tape_dir/file.nc /lustre_dir
Retrieval of a whole directory¶
pftp¶
$ cd /lustre_dir
$ pftp
ftp> cd /tape_dir
ftp> prompt
ftp> mget *
ftp> quit
slk¶
$ slk retrieve /tape_dir/ /lustre_dir
All files located in /tape_dir
are retrieved. No recursive retrieval of namespaces located in /tape_dir
is performed. If you wish to do a recursive retrieval of the whole content of /tape_dir
, just add the -R
option as follows:
$ slk retrieve -R /tape_dir/ /lustre_dir
Note
You do not need to add *
to the source path as would be necessary for cp
(cp /tape_dir/* /lustre_dir
). If you add *
to the source path in the slk retrieve
call it will fail (slk retrieve /tape_dir/* /lustre_dir
).
Note
The notation of the source diretory in slk retrieve
is similar to rsync
: slk retrieve /tape_dir/ /lustre_dir
is different to slk retrieve /tape_dir /lustre_dir
(no slash). Only the folders content is retrieved when /
is appended (like cp /tape_dir/* /lustre_dir
). The source folder itself (with its content) is retrieved when /
is not appended (like cp /tape_dir/* /lustre_dir/`basename /tape_dir`
).
Listing the content of a folder in the archive¶
pftp¶
$ pftp
ftp> cd /tape_dir
ftp> ls
200 PORT command successful.
150 Opening ASCII mode data connection for file list.
a.nc
b.nc
c.nc
d.nc
c.nc
226 Transfer complete.
312 bytes received in 0.0054 seconds (0.056 MBytes/sec)
ftp> quit
slk¶
$ slk list /tape_dir
-rw-r--r--- k204221 bm0146 4 04 Feb 2021 a.nc
-rw-r--r--- k204221 bm0146 6.5K 04 Feb 2021 b.nc
-rw-r--r--- k204221 bm0146 10 07 Feb 2021 c.nc
-rw-r--r--- k204221 bm0146 16.1K 01 Feb 2021 d.nc
-rw-r--r--- k204221 bm0146 1.2M 04 Feb 2021 e.nc