Switching: pftp to slk

file version: 28 February 2022

current slk version: 3.3.21

Some common pftp workflows and their counterparts using slk are listed below to simplify switching from pftp to slk. A general description of slk is given in StrongLink Command Line Interface (slk) and additional example calls are provided in slk Usage Examples.

Nomenclature

We use some abbreviated file and folder names in the code examples for simpler readability. These are

• /lustre_dir: a directory path on the lustre file system (e.g /work/bm0146/k204221)

• /tape_dir: a directory path in the tape archive (e.g today /hpss/arch/bm0146/k204221)

• file.nc: a netCDF file

Login

pftp

$ pftp
...
ftp>

slk

Login to another shell like to the ftp shell via pftp is not necessary. Instead, the user has to generate a login token every 30 days. This procedure is comparable to the Kerberos authentication. The slk commands are run on the normal Linux shell.

$ slk login
Username: XYZ
Password:
Login Successful

Note

slk stores the login token in the home directory of each user (~/.slk/config.json). By default, this file can only be accessed by the respective user (default permissions: -rw-------/600). However, users should be careful when doing things like chmod 755 * in their home directory so that other users cannot access the token. If you assume that your slk login token has been compromized please contact support@dkrz.de .

Archival of a single file

pftp

$ cd /lustre_dir
$ pftp
# possibly `mkdir tape_dir`
ftp> cd /tape_dir
ftp> put file.nc
ftp> quit

slk

$ slk archive /lustre_dir/file.nc /tape_dir

Archival of a whole directory

pftp

$ cd /lustre_dir
$ pftp
# possibly `mkdir /tape_dir`
ftp> cd /tape_dir
ftp> prompt
ftp> mput *
ftp> quit

slk

$ slk archive /lustre_dir /tape_dir

All files located in /lustre_dir are archived. No recursive archival of directories located in /lustre_dir is performed. If you wish to do a recursive archival of the whole folder tree below /lustre_dir, just add the -R option as follows:

$ slk archive -R /lustre_dir /tape_dir

Note

You do not need to add * to the source path as would be necessary for cp (cp /lustre_dir/* /tape_dir/`basename /lustre_dir`). If you add * to the source path in the slk archive call (slk archive /lustre_dir/* /tape_dir) then you would archive all sub-directories with their content. The sub-directories/namespaces are automatically created in StrongLink. If you have sub-folders 1900 until 1999 (one output folder per model year) and you only want to archive the 1980s then you simply do slk archive /lustre_dir/198? /tape_dir.

Retrieval of a single file

pftp

$ cd /lustre_dir
$ pftp
ftp> cd /tape_dir
ftp> get file.nc
ftp> quit

slk

$ slk retrieve /tape_dir/file.nc /lustre_dir

Retrieval of a whole directory

pftp

$ cd /lustre_dir
$ pftp
ftp> cd /tape_dir
ftp> prompt
ftp> mget *
ftp> quit

slk

$ slk retrieve /tape_dir/ /lustre_dir

All files located in /tape_dir are retrieved. No recursive retrieval of namespaces located in /tape_dir is performed. If you wish to do a recursive retrieval of the whole content of /tape_dir, just add the -R option as follows:

$ slk retrieve -R /tape_dir/ /lustre_dir

Note

You do not need to add * to the source path as would be necessary for cp (cp /tape_dir/* /lustre_dir). If you add * to the source path in the slk retrieve call it will fail (slk retrieve /tape_dir/* /lustre_dir).

Note

The notation of the source diretory in slk retrieve is similar to rsync: slk retrieve /tape_dir/ /lustre_dir is different to slk retrieve /tape_dir /lustre_dir (no slash). Only the folders content is retrieved when / is appended (like cp /tape_dir/* /lustre_dir). The source folder itself (with its content) is retrieved when / is not appended (like cp /tape_dir/* /lustre_dir/`basename /tape_dir`).

Listing the content of a folder in the archive

pftp

$ pftp
ftp> cd /tape_dir
ftp> ls
200 PORT command successful.
150 Opening ASCII mode data connection for file list.
a.nc
b.nc
c.nc
d.nc
c.nc
226 Transfer complete.
312 bytes received in 0.0054 seconds (0.056 MBytes/sec)
ftp> quit

slk

$ slk list /tape_dir
-rw-r--r--- k204221    bm0146       4   04 Feb 2021   a.nc
-rw-r--r--- k204221    bm0146    6.5K   04 Feb 2021   b.nc
-rw-r--r--- k204221    bm0146      10   07 Feb 2021   c.nc
-rw-r--r--- k204221    bm0146   16.1K   01 Feb 2021   d.nc
-rw-r--r--- k204221    bm0146    1.2M   04 Feb 2021   e.nc