FAQ¶
v1.34, 19 April 2022
Table of Contents
-
-
Do I have to provide my login credentials each time I use the command line tool?
Do I have write access to the archive from outside the DKRZ?
Will the new system be available as a Globus endpoint for external transfers?
Does StrongLink automatically check the integrity of archived and retrieved files?
Do I manually need to check the integrity of archived and retrieved files?
Is there an option to continue archiving if it was interrupted?
Does any command exist for deleting files immediately from /work in case of successful archival?
Is it possible to archive into my existing folder structure created on HPSS?
How do I automatically/non-interactively check whether I own a valid slk login token?
Can I provide a file list to “slk archive” such as “-T” for “tar”?
Can a user run multiple archival and retrieval requests at a time?
Are there requirements on the file size for the tape archival?
I am member of a project but cannot access this projects data?
Why do I get “Exception …: lateinit property websocket has not been initialized”?
-
Can a user influence if data is written into the HSM cache or onto tape?
How do I determine the username of a DKRZ user when I have her/his id (uid)?
How do I determine the group name of a DKRZ group when I have its id (gid)?
Can the search ID of slk search be captured by a shell variable?
Is the metadata of files within zip/tar files evaluated/ingested?
Is it possible to create symlinks between lustre_path/files and tape_path/files?
Is it possible to use listems to list files that were archived with packems on the HPSS?
Is it possible to use unpackems to retrieve files that were archived with packems on the HPSS?
Can you work directly with files in the archive (e.g. with Python)?
Terminal cursor disapears after stopping a slk command. How to get it back?
Is a file stored in the HSM cache already or exclusively on tape?
-
General information about the HSM system¶
What does HSM mean?¶
What type of HSM system is used at DKRZ?¶
Why did DKRZ get a new system?¶
What are the main differences compared to the old system?¶
Is the new HSM system accessbile via pftp?¶
pftp
was replaced by a new command line tool slk
. Additionally, a command line tool slk_helpers
is provided which has some features that slk
is lacking. slk
is developed and maintained by StrongBox Data Solutions, whereas slk_helpers
are developed and maintained at DKRZ.Would it be possible/desirable to use only one command for slk and slk_helpers main classes?¶
Data Migration¶
When did the new HSM system go online?¶
Are my archived data available on the new system?¶
How do I find out whether I have data from DXUL that have to be copied manually?¶
How do I access DXUL data after the HPSS is shut down?¶
How to proceed if I still have DXUL data that need to be kept?¶
Training, Questions and Adaption of Workflows¶
Has there been an introduction session to the new HSM system and will there be such sessions in future?¶
slk
were presented there. A recording of the Tech Talk available on YouTube: https://www.youtube.com/watch?v=JtmelPQ3ypw. We offered a HSM Q&A session each Thursday 11:30 - 12:30 AM from 30 Sept to end of December 2021. We plan a second TechTalk when the system is fully functional.Where can I find written documentation about the new HSM system?¶
Why is no exact time schedule for training and migration published yet?¶
Who do I contact when I have questions or issues regarding the new HSM system and its usage?¶
Archiving and Retrieval¶
How do I interact with the new system?¶
slk
. Some features are deactivated for now. Additionally, the slk
is missing a few small but very useful features. Therefore, a tool called slk_helpers
was written at DKRZ to add these features. Details on these two tools are provided in the HSM Documentation at https://docs.dkrz.de .Where can I use slk
and slk_helpers
?¶
slk
/slk_helpers
are be installed as module slk on all mistral and lenvate nodes. On the login nodes (mlogin100
to mlogin108
; levante1
to levante7
), please use slk archive
and slk retrieve
only for a few small files in MB-size or below. For details on further restrictions and recommendations please have a look into the documentation (section On which nodes to run slk on page Getting Started with slk).Can I still use pftp to interact with the new HSM system?¶
How do I login to the HSM system?¶
Does Kerberos authentication work on the new HSM system?¶
Do I have to provide my login credentials each time I use the command line tool?¶
Can I use the command line tool non-interactively?¶
Can I access archived data from outside the DKRZ?¶
Do I have write access to the archive from outside the DKRZ?¶
slk
is not made for data transfer via the internet. An exception are institutions who have direct network connection to DKRZ and currently have access to the HPSS via pftp. Users from these institutions will be able to use slk.Will the new system be available as a Globus endpoint for external transfers?¶
Does the tape quota (/arch, /doku), which was assigned to my computing time project, remain unchanged?¶
What happens if I archive a soft link?¶
How do I create directories in the HSM?¶
slk mkdir
does not exist but the slk_helpers
provide it. Use slk_helpers mkdir /ex/am/ple/dir
if /ex/am/ple
already exists and you only want to create dir
. If you want to create several nested folders (like mkdir -p
does) please use slk_helpers mkdir -R /ex/am/ple/dir
. If you do not want to use the slk_helpers
and only slk
, please do as follows: create empty directories locally, fill them with non-empty dummy files and archive them via slk archive -R
. An example for this process is given in the Use Case section of the HSM documentationDoes StrongLink automatically check the integrity of archived and retrieved files?¶
slk_helpers checksum RESOURCE
and compare them against the checksum of the local copies of the respective files.Files may be partly archived when slk archive
is killed while these files are archived. Killed means events like: timeout of SLURM job script, manual termination via CTRL + C, process killed, timeout/disconnect of ssh session. Such partially archived files remain in the StrongLink cache until they are deleted. They are not written onto tape and no checksum is calculated. Partial files that were archived before mid December 2021 might have a checksum and might have been written onto tape.
Please also have a look into Getting Started for details.
Do I manually need to check the integrity of archived and retrieved files?¶
Is there an option to continue archiving if it was interrupted?¶
slk archive
will not upload files a second time after its restart but only those files that are not already present in the target folder. If only a part of the file was uploaded at the time of interruption, then the upload of the whole file will be restarted when the archival process is restarted. After stopping an archival process a file fragment will remain in the archive. Either the archival process has to be resumed (the fragment will be overwritten then) or the file has to be deleted manually. File fragments have basic metadata attached to them but no checksums. When you can get a checksum for a particular file from StrongLink (slk_helpers checksum ...
) then the file was archived successfully.Does any command exist for deleting files immediately from /work in case of successful archival?¶
Is it possible to archive into my existing folder structure created on HPSS?¶
/hpss
was dropped.Is there a “double” storage feature as for HPSS?¶
What does “namespace”, “global namespace” or “gns” mean?¶
How do I automatically/non-interactively check whether I own a valid slk login token?¶
slk
does not provide a command that returns the status of the login tokes as true
/false
, valid
/invalid
or similar does not exist yet. But, you can check the validity of your login token via slk_helpers session
. If you do not want to use the slk_helpers
but check the status of the login token anyway, please use one of the following two commands:# command 1:
$ slk list /dummy_input < /dev/null > /dev/null 2>&1
# command 2:
$ test `jq .expireDate ~/.slk/config.json` -gt `date +%s`
$?
will be 0
if login token is valid and 1
if not. Thanks to Karl-Hermann Wieners for the first command.
You need to have the program jq
available for the second command. jq
is installed in /sw/rhel6-x64/devtools/jq-1.6-gcc48/bin/jq
. You might add /sw/rhel6-x64/devtools/jq-1.6-gcc48/bin/
to your PATH
or set an alias.
Is my slk login token still valid?¶
How to I check for how long my login token is still valid?¶
slk_helpers session
will print the expiration date. Alternatively, the date/time until when the login token is stored in the slk config file (~/.slk/config.json
). The key is expirationDate
. You might open the config file with a text editor or print its content with tools like cat
, less
or jq
.jq .expireDate ~/.slk/config.json
You need to have the program jq
available. jq
is installed in /sw/rhel6-x64/devtools/jq-1.6-gcc48/bin/jq
on mistral and in /usr/bin/jq
on levante. On mistral, you might add /sw/rhel6-x64/devtools/jq-1.6-gcc48/bin/
to your PATH
or set an alias.
Can I provide a file list to “slk archive” such as “-T” for “tar”?¶
Can a user run multiple archival and retrieval requests at a time?¶
mlogin10X
, mistralppY
, levanteZ
) or on shared nodes we suggest to run only one slk call per user and node to avoid memory issues. For details please see section On which nodes to run slk on page Getting Started with slk of the documentation.Where on mistral and levante should I run slk?¶
slk
uses much CPU time and memory. Therefore, slk archive
and slk retrieve `` should only be used for small amounts of data on the login nodes (``mlogin10X
, levanteY
). For large amounts of data, we suggest to use the compute
/compute2
. For details and alternatives please see section On which nodes to run slk on page Getting Started with slk of the documentation.How does slk archive the files: does it tar them itself (similar to packems) or should we tar the files before hand?¶
slk
does not packs/tar files. Metadata from netCDF files is automatically imported into the StrongLink database to simplify search and retrieval later on. Direct archiveal of nc-files is preferable with respect to the metadata import feature. However, many small files are bad for tape performance and might cost additional storage space (see Storage options and quota. Therefore, the usage of packems is reasonable in the case of large amount of very small files.Are there requirements on the file size for the tape archival?¶
I am member of a project but cannot access this projects data?¶
Please login again via slk login
. For details please see group memberships of user updated on login.
Why do I get “Exception …: lateinit property websocket has not been initialized”?¶
When running slk archive
with the argument --streams N
please do only use values between 1
and 4` for N
. For details please see slk archive: Exception …: lateinit property websocket has not been initialized.
My slk archive seems to hang. What should I do?¶
Please check whether /home
is hanging. If /home
is hanging, slk
cannot access its login token and cannot write into its log. Therefore, slk
hangs when /home
is hanging.
Please check whether you are archiving a file of 0 Byte size. Details on the Known Issues page.
Additional features¶
Which new features does the HSM System provide?¶
slk search
based on these extended metadata. Currently, slk search
is deactivated. Please use slk_helpers search_limited
for the time being.From which file types are extended metadata harvested?¶
Which metadata fields are harvested from netCDF files?¶
Is there a python interface available?¶
pyslk
. It is installed in python3/2021.01-gcc-9.1.0
and python3/unstable
on mistral and in python3/2022.01-gcc-11.2.0
on levante. The slk
module as to be loaded when pyslk
is used. See for details please see https://hsm-tools.gitlab-pages.dkrz.de/pyslk .Is it possible to use slk chmod and slk group (=chgrp) commands recursively by the user?¶
-R
to apply these commands recursively.Are the search IDs user specific?¶
slk list SEARCH_ID
or retrieval of slk retrieve SEARCH_ID ...
depends on the read permissions of the executing user.How long are the search IDs stored?¶
Is a search ID automatically updated when new files are archived which match the original search query?¶
slk list SEARCH_ID
will show todays sizes of files covered by the search ID SEARCH_ID
. Files that first matched the a search query are still listed by slk list
even if they no longer match the original search query. This might happen if a file is renamed.What does “RQL” mean?¶
What is the “StrongLink Query Language”?¶
slk search
(deactivated; please use slk_helpers search_limited
).Is there any possibility to move around in the filesystem with something like the cd command?¶
slk
does not start it own shell like pftp
or pure ftp
do. It rather works like scp
.When slk list shows a file with “-” (not “t”) which means it exists at the cache: Does that mean it is not yet on the tape?¶
For a better overview of the archived files, Is there a possibility to list only folders, not all files?¶
slk list
with a specific namespace path, it shows all the files and namespaces in that specific namespace. If you use the -R
flag, it shows all the files and namespaces recursively (like ls -R
). So if you want a clean overview, excluding -R
would be the way. You might use slk list GNS_PATH | grep -E "^d"
to print only folders.Is it possible to remove files from the archive?¶
slk delete
for removing files and slk delete -R
for removing namespaces.How to print the version of slk?¶
slk version
to print the version of slk
. A --version
flag or similar does not exist.How to search non-recursively in a namespace?¶
slk search
cannot search non-recursively in a namespace provided via path
. As workaround, please get the object id of the particular namespace via slk_helpers exists
and, then in your search query, use it as value for the search field resources.parent_id
(see slk Usage Examples)
Is it possible to move files within the archive?¶
slk move
for move a file or namespace from one namespace to another. Absolute paths have to be used: slk move /old_path/file.nc /new_path
. Renaming cannot be done with slk move
. I.e. this does not work: slk move /old_path/file.nc /new_path/new_file_name.nc
. Please use slk rename
for renaming operations.Is it possible to rename files within the archive?¶
slk rename
to rename a file or namespace. slk rename
cannot be applied on multiple files/namespacesHow do I tag a folder with metadata?¶
Tagging folders with metadata is not possible in the moment.
How do I tag an individual file with metadata?¶
Tagging individual files with metadata is not possible in the moment.
slk search does not find any resources although resources exist that seem to match the query¶
Example command:¶
$ slk search '{"resources.posix_uid": "25301"}'
Search continuing. .....
Search ID: 216
Reason:¶
The query parser does not recognize when a wrong variable type is used. resources.posix_uid
is of type integer
and not string
. Providing the wrong data types leads to 0 found results.
Solution:¶
Write 25301
(integer) instead of "25301"
(string) in the search query.
slk search '{"resources.posix_uid": 25301}'
Search continuing. ..... Search continuing. .....
Search ID: 217
Error: slk search yields RQL parse error¶
Example command and error:¶
$ slk search "{\"resources.size\":{\"$gt\": 1048576}}"
ERROR: Search failed. Reason: RQL parse error: No period found in collection field name ().
Reason:¶
The $
in front of the gt
was not escaped. Therefore, $gt
is interpreted as environment variable by the shell before the query is handed to the slk
. In most situations, no environment variable gt
is defined leading to an empty string. If the query were surrounded by '
as delimiter and not by "
then the $gt
would not have been interpreted.
The above call of slk search
as interpreted by the shell looks like
$ slk search "{\"resources.size\":{\"\": 1048576}}"
Solution:¶
Either: use '
as delimiter of your search query instead of "
to prevent operators starting with $
to be evaluated by your shell
Or: escape $
’s in front of query operators by \
when you use "
as delimiters of the query string.
'{"resources.size":{"$gt": 1048576}}'
"{\"resources.size\":{\"\$gt\": 1048576}}"
Note
In some situations it might be very useful to use "
as delimiter for your queries – e.g. if environment variables are part of your query.
$ export file_size=1048576
$ slk search "{\"resources.size\":{\"\$gt\": $file_size}}"
Advanced Technical Aspects¶
Can a user influence if data is written into the HSM cache or onto tape?¶
How much time does a file stay on the cache?¶
How fast can be read from the HSM?¶
slk retrieve
will be idle until a tape drive is free.Do GIGA-files still exists in StrongLink?¶
slk search
tool followed by slk list
to generate a list of all files of your project:# NOTE: PATH might be different in final HSM setup
$ SEARCH_ID=`slk search "{\"path\":{\"\$gte\":\"/ex/am/ple/arch/bm0146\"}}"`
# OR
$ SEARCH_ID=`slk search "{\"resources.posix_gid\":1076}"`
$ slk list $SEARCH_ID
How do I determine the id (uid) of a DKRZ user?¶
# get your user id
$ id -u
# get the id of any user
$ id USER_NAME -u
# get the id of any user
$ getent passwd USER_NAME
# OR
$ getent passwd USER_NAME | awk -F: '{ print $3 }'
How do I determine the id (gid) of a DKRZ group?¶
# get group ID and group members
$ getent group GROUP_NAME
# OR
$ getent group GROUP_NAME | awk -F: '{ print $3 }'
# get groups and their ids of all groups of which member you are
$ id
How do I determine the username of a DKRZ user when I have her/his id (uid)?¶
# get the name of a user with uid USER_ID
$ getent passwd USER_ID
# OR
$ getent passwd USER_ID | awk -F: '{ print $1 }'
How do I determine the group name of a DKRZ group when I have its id (gid)?¶
# get group name of a groupd with gid GROUP_ID
$ getent group GROUP_ID
# OR
$ getent group GROUP_ID | awk -F: '{ print $1 }'
How do I determine the MIME type of a file?¶
file --mime-type FILE
or file -b --mime-type FILE
to determine the MIME type on the Linux shell. Please be aware that different tools determine the MIME type differently (i.e. by file header or by file extension) and MIME type databases might differ. It might be better not to search for a specific MIME type but for a particular file extension – e.g. via {"resources.name": {"$regex": ".*nc$"}}
. StrongLink allocates the MIME type application/x-netcdf
to netCDF files.Can the search ID of slk search be captured by a shell variable?¶
slk search
(currently deactivated; please use slk_helpers search_limited
) does not provide this feature out of the box. Currently (might change in future versions), the search ID is printed in columns >= 12 of the second row of the text output of slk search
. We can use tail
and sed
to get the second line and extract a number or use tail
and cut
to get the second line and drop the first 11 characters. Example:# normal call of slk search
$ slk search '{"resources.posix_uid": 23501}'
Search continuing. .....
Search ID: 466
# get ID using sed:
$ search_id=`slk search '{"resources.posix_uid": 23501}' | tail -n 1 | sed 's/[^0-9]*//g'`
$ echo $search_id
470
# get ID by dropping first 11 characters of the second line
$ search_id=`slk search '{"resources.posix_uid": 23501}' | tail -n 1 | cut -c12-20`
$ echo $search_id
471
# use awk pattern matching to get the correct line and correct column
$ search_id=`slk search '{"resources.posix_uid": 25301}' | awk '/Search ID/ {print($3)}'`
$ echo $search_id
507
Note
This is an example for bash
. When using csh
, you need to prepend set `` in front of the assignments of the shell variables: ``set search_id=...
.
Is the metadata of files within zip/tar files evaluated/ingested?¶
Is it possible to create symlinks between lustre_path/files and tape_path/files?¶
Does the packems package work with the new HSM system?¶
slk retrieve
is still limited. Please have a look into the packems manual for details and usage of packems: https://code.mpimet.mpg.de/projects/esmenv/wiki/Packems.Is it possible to use listems to list files that were archived with packems on the HPSS?¶
slk retrieve
is activated for interactive use, all files archived with packems onto the HPSS can be listed with listems. Currently, slk retrieve
is not fully available and listems works only with INDEX.txt
files stored on the lustre filesystem.Is it possible to use unpackems to retrieve files that were archived with packems on the HPSS?¶
slk retrieve
will be fully activated, all files archived with packems onto the HPSS can be retrieved with unpackems. Currently, slk retrieve
is not fully available. Therefore, individual INDEX.txt
and *.tar
files need to be retrieved manually.Can you work directly with files in the archive (e.g. with Python)?¶
Terminal cursor disapears after stopping a slk command. How to get it back?¶
If a slk command with a progress bar is canceled by the user, the shell cursor might disappear. One can make it re-appear by (a) running reset
or (b) starting vim
and leaving it directly (:q!
).
Is a file stored in the HSM cache already or exclusively on tape?¶
Solution a: In the output of slk list
, please check the 11th character of the first column (permissions string). If this character is t
then the file is exclusively stored on tape. If it is a -
then the file is available from the HSM cache.
Solution b: Use slk_helpers iscached RESOURCE_PATH
to check whether a file available from the HSM cache (exit code is 0
) or not (exit code is 1
).
Common issues¶
Please see the extra page Known Issues
Changelog¶
v1.34, 19 April 2022¶
modified: Where can I use slk and slk_helpers?
modified: Can a user run multiple archival and retrieval requests at a time?
modified: What is the “StrongLink Query Language”?
modified: Can the search ID of slk search be captured by a shell variable?
modified: Does the packems package work with the new HSM system?
modified: Is it possible to use listems to list files that were archived with packems on the HPSS?
modified: Is it possible to use unpackems to retrieve files that were archived with packems on the HPSS?
v1.33, 30 March 2022¶
modified: Is there a python interface available?
modified: Can a user run multiple archival and retrieval requests at a time?
new: I am member of a project but cannot access this projects data?
new: Why do I get “Exception …: lateinit property websocket has not been initialized”?
new: Is a file stored in the HSM cache already or exclusively on tape?
new: slk search does not find any resources although resources exist that seem to match the query
v1.32, 28 February 2022¶
removed: Does the tape archive hardware also change?
removed: When did the HPSS go offline? / When does the HPSS go offline?
modified: How do I find out whether I have data from DXUL that have to be copied manually?
modified: Where can I use slk and slk_helpers?
modified: How do I login to the HSM system?
modified: Does StrongLink automatically check the integrity of archived and retrieved files?
modified: Do I manually need to check the integrity of archived and retrieved files?
modified: How to I check for how long my login token is still valid?
modified: Are there requirements on the file size for the tape archival?
modified: From which file types are extended metadata harvested?
modified: Is there a python interface available?
modified: How fast can be read from the HSM?
modified: Does the packems package work with the new HSM system?
modified: Is it possible to use listems to list files that were archived with packems on the HPSS?
modified: Is it possible to use unpackems to retrieve files that were archived with packems on the HPSS?
renamed: from Where on mistral and levante should I run slk? to Where on mistral should I run slk?
v1.31, 11 February 2022¶
v1.30, 06 December 2021¶
removed content of section Common Issues (moved to page Known Issues)
removed: error “conflict with jdk/…” when the slk module is loaded (moved to page Known Issues)
removed: slk needs a specific Java version (moved to page Known Issues)
removed: slk search yields RQL parse error (moved to page Known Issues)
removed: slk login asks me to provide a hostname and/or a domain (moved to page Known Issues)
removed: Session key has expired (moved to page Known Issues)
removed: Login Unsuccessful - Incorrect Credentials (moved to page Known Issues)
removed: Archival fails and Java NullPointerException in the log (moved to page Known Issues)
renamed: from Terminal cursor disapears after stopping a slk command to Terminal cursor disapears after stopping a slk command. How to get it back?
v1.29, 18 November 2021¶
modified: Session key has expired
v1.28, 12 November 2021¶
modified: How do I create directories in the HSM?
v1.27, 11 November 2021¶
renamed: from What type of HSM system will be installed? to What type of HSM system is used at DKRZ?
renamed (and modified): from Why is DKRZ getting a new system? to Why did DKRZ get a new system?
modified: What are the main differences compared to the old system?
renamed: from Will the new HSM system be accessbile via pftp? to Is the new HSM system accessbile via pftp?
modified: Does the tape archive hardware also change? (removed in FAQ version 1.32)
removed: Will there be a continuous changeover from HPSS to StrongLink HSM?
renamed (and modified): from When does the new HSM system go online? to When did the new HSM system go online?
renamed: from When does the HPSS go offline? to When did the HPSS go offline? (removed in FAQ version 1.32)
removed: When will an exact time schedule for the migration be published?
renamed (and modified): from Will all my archived data be available on the new system? to Are my archived data available on the new system?
removed: Where do I find data from the DXUL archive now?
modified: How do I access DXUL data after the HPSS is shut down?
modified: How to proceed if I still have DXUL data that need to be kept?
removed: What do I do with simulation results during the downtime between HPSS going offline and StrongLink going online?
renamed (and modified): from Will there be an introduction session to the new HSM system and its usage? to Has there been an introduction session to the new HSM system and will there be such sessions in future?
modified: Why is no exact time schedule for training and migration published yet?
modified: Where can I use slk and slk_helpers?
modified: Do I have to provide my login credentials each time I use the command line tool?
removed: How do I use the HSM/StrongLink test system?
modified: Do I manually need to check the integrity of archived and retrieved files?
modified: Can a user run multiple archival and retrieval requests at a time?
modified: Are there requirements on the file size for the tape archival?
renamed: from From which file types is extended metadata harvested? to From which file types are extended metadata harvested?
modified: Which metadata fields are harvested from netCDF files?
removed: Why does slk search show more search results than slk list lists for this search id?
modified: Is the metadata of files within zip/tar files evaluated/ingested?
modified: Is it possible to create symlinks between lustre_path/files and tape_path/files?
renamed: from Will it be possible to use listems to list files that were archived with packems on the HPSS? to Is it possible to use listems to list files that were archived with packems on the HPSS?
renamed: from Will it be possible to use unpackems to retrieve files that were archived with packems on the HPSS? to Is it possible to use unpackems to retrieve files that were archived with packems on the HPSS?
v1.26, 01 November 2021¶
removed: Will I be able to see how the new HSM system will look like before it becomes productive?
removed: Will DKRZ users be able to test their archiving workflows before the new system goes online?
modified: When does the new HSM system go online?
modified: Why is no exact time schedule for training and migration published yet?
modified: Can I still use pftp to interact with the new HSM system?
renamed: from How will I interact with the new system? to How do I interact with the new system?
modified: Do I have write access to the archive from outside the DKRZ?
renamed: from Will it be possible to archive into my existing folder structure created on HPSS? to Is it possible to archive into my existing folder structure created on HPSS?
renamed: from Will there be a “double” storage feature as for HPSS? to Is there a “double” storage feature as for HPSS?
renamed: from From which file types will extended metadata be harvested? to From which file types is extended metadata harvested?
renamed: from Which metadata fields will be harvested from netCDF files? to Which metadata fields are harvested from netCDF files?
modified: Is there a python interface available?
renamed: from Will the packems package work with the new HSM system? to Does the packems package work with the new HSM system?
v1.25, 27 October 2021¶
modified: When does the new HSM system go online?
modified: Why is no exact time schedule for training and migration published yet?
modified: Why does slk search show more search results than slk list lists for this search id? (removed in FAQ version 1.27)
modified: Can the search ID of slk search be captured by a shell variable?
v1.24, 23 October 2021¶
v1.23, 15 October 2021¶
modified: When does the new HSM system go online?
modified: When does the HPSS go offline? (removed in FAQ version 1.32)
modified: Why is no exact time schedule for training and migration published yet?
v1.22, 08 October 2021¶
modified: When does the new HSM system go online?
modified: When does the HPSS go offline? (removed in FAQ version 1.32)
modified: Why is no exact time schedule for training and migration published yet?
v1.21, 01 October 2021¶
new: Archival fails and Java NullPointerException in the log
v1.20, 29 September 2021¶
v1.19, 20 September 2021¶
changed title of FAQ
corrected FAQ’s Changelog
v1.18, 17 September 2021¶
added cross-references
minor layout changes
v1.17, 17 September 2021¶
modified: When does the new HSM system go online?
modified: When does the HPSS go offline? (removed in FAQ version 1.32)
modified: Why is no exact time schedule for training and migration published yet?
modified: Who do I contact when I have questions or issues regarding the new HSM system and its usage?
v1.16, 17 August 2021¶
modified: When does the new HSM system go online?
modified: When does the HPSS go offline? (removed in FAQ version 1.32)
modified: Why is no exact time schedule for training and migration published yet?
v1.15, 30 July 2021¶
modified: When does the new HSM system go online?
modified: When does the HPSS go offline? (removed in FAQ version 1.32)
modified: Why is no exact time schedule for training and migration published yet?
new: How do I use the HSM/StrongLink test system? (removed in FAQ version 1.27)
new: Session key has expired
new: Login Unsuccessful - Incorrect Credentials
v1.14, 12 July 2021¶
new: Would it be possible/desirable to use only one command for slk and slk_helpers main classes?
modified: When does the new HSM system go online?
modified: When does the HPSS go offline? (removed in FAQ version 1.32)
modified: Will there be an introduction session to the new HSM system and its usage?
modified: Will DKRZ users be able to test their archiving workflows before the new system goes online?
modified: Why is no exact time schedule for training and migration published yet?
new: Do I have write access to the archive from outside the DKRZ?
new: Will the new system be available as a Globus endpoint for external transfers?
modified Does StrongLink automatically check the integrity of archived and retrieved files?
new: Can I provide a file list to “slk archive” such as “-T” for “tar”?
new: Can a user run multiple archival and retrieval requests at a time?
new: Are there requirements on the file size for the tape archival?
modified: Why does slk search show more search results than slk list lists for this search id? (removed in FAQ version 1.27)
new: Is there any possibility to move around in the filesystem with something like the cd command?
new: Can you work directly with files in the archive (e.g. with Python)?
v1.13, 29 June 2021¶
modified: When does the new HSM system go online?
modified: When does the HPSS go offline? (removed in FAQ version 1.32)
modified: Why is no exact time schedule for training and migration published yet?
v1.12, 08 June 2021¶
modified: How will I interact with the new system?
modified: Where can I use slk and slk_helpers?
modified: How do I create directories in the HSM?
modified: Does StrongLink automatically check the integrity of archived and retrieved files?
modified: Do I manually need to check the integrity of archived and retrieved files?
modified: Is there an option to continue archiving if it was interrupted?
modified: How do I automatically/non-interactively check whether I own a valid slk login token?
modified: How to I check for how long my login token is still valid?
modified: Why does slk search show more search results than slk list lists for this search id? (removed in FAQ version 1.27)
modified: Can the search ID of slk search be captured by a shell variable?
v1.11, 06 May 2021¶
new: How to I check for how long my login token is still valid?
modified: Will I be able to see how the new HSM system will look like before it becomes productive?
modified: Will there be an introduction session to the new HSM system and its usage?
modified: Will DKRZ users be able to test their archiving workflows before the new system goes online?
modified: Why is no exact time schedule for training and migration published yet?
modified: How do I login to the HSM system?
modified: Can I use the command line tool non-interactively?
modified: How do I create directories in the HSM?
modified: What does “namespace”, “global namespace” or “gns” mean?
modified: Are the search IDs user specific?
modified: How do I automatically/non-interactively check whether I own a valid slk login token?
rephrased question: What do I do with simulation results during the downtime between HPSS going offline and StrongLink going online? (removed in FAQ version 1.27)
rephrased question: How do I determine the id (uid) of a DKRZ user?
rephrased question: How do I determine the id (gid) of a DKRZ group?
rephrased question: How do I determine the username of a DKRZ user when I have her/his id (uid)?
rephrased question: How do I determine the group name of a DKRZ group when I have its id (gid)?
rephrased question: How do I determine the MIME type of a file?
v1.10, 23 April 2021¶
new: Why does slk search show more search results than slk list lists for this search id? (removed in FAQ version 1.27)
modified: When does the new HSM system go online?
modified: When does the HPSS go offline? (removed in FAQ version 1.32)
modified: Will there be an introduction session to the new HSM system and its usage?
modified: Will DKRZ users be able to test their archiving workflows before the new system goes online?
removed: slk is called in a directory in which the user has no write permissions
v1.09, 06 April 2021¶
modified: When does the new HSM system go online?
modified: When does the HPSS go offline? (removed in FAQ version 1.32)
modified: Where do I find data from the DXUL archive now? (removed in FAQ version 1.27)
modified: Will there be an introduction session to the new HSM system and its usage?
modified: Why is no exact time schedule for training and migration published yet?
modified: Can I use the command line tool non-interactively?
modified: Is it possible to use slk chmod and slk group (=chgrp) commands recursively by the user?
v1.08, 12 March 2021¶
v1.07, 10 March 2021¶
modified: Why is DKRZ getting a new system?
modified: What are the main differences compared to the old system?
modified: Where can I find written documentation about the new HSM system?
modified: Why is no exact time schedule for training and migration published yet?
modified: How will I interact with the new system?
modified: How do I login to the HSM system?
modified: Do I have to provide my login credentials each time I use the command line tool?
modified: Does StrongLink automatically check the integrity of archived and retrieved files?
modified: Do I manually need to check the integrity of archived and retrieved files?
modified: Can the search ID of slk search be captured by a shell variable?
modified: Is the metadata of files within zip/tar files evaluated/ingested?
modified: error “conflict with jdk/…” when the slk module is loaded
v1.06, 08 March 2021¶
modified: How do I create directories in the HSM?
new: What to do with simulation results during the downtime between HPSS going offline and StrongLink going online? (removed in FAQ version 1.27)
new: Is there an option to continue archiving if it was interrupted?
new: Does any command exist for deleting files immediately from /work in case of successful archival?
new: Will it be possible to archive into my existing folder structure created on HPSS?
new: What does “*namespace*”, “*global namespace*” or “*gns*” mean?
new: Is it possible to use chmod and chgrp commands recursively by the user?
modified: How do I get the username of a DKRZ user when I have her/his id (uid)?
modified: How do I get the group name of a DKRZ group when I have its id (gid)?
new: Can the search ID of slk search be captured by a shell variable?
new: Is the metadata of files within zip/tar files evaluated/ingested?
new: Is it possible to create symlinks between lustre_path/files and tape_path/files?
new: error “conflict with jdk/…” when the slk module is loaded
new: slk login asks me to provide a hostname and/or a domain
v1.05, 23 February 2021¶
new: Does StrongLink automatically check the integrity of archived and retrieved files?
new: Do I manually need to check the integrity of archived and retrieved files?
new: How do I get the username of a DKRZ user when I have her/his id (uid)?
new: How do I get the group name of a DKRZ group when I have its id (gid)?
new section Common Issues
v1.04, 22 February 2021¶
v1.03, 18 February 2021¶
modified: When does the new HSM system go online?
modified: When does the HPSS go offline? (removed in FAQ version 1.32)
modified: Where can I find written documentation about the new HSM system?
v1.02, 12 February 2021¶
modified: When does the new HSM system go online?
new: When will an exact time schedule for the migration be published? (removed in FAQ version 1.27)
new: Will I be able to see how the new HSM system will look like before it becomes productive?
modified: Will there be an introduction session to the new HSM system and its usage?
new: Why is no exact time schedule for training and migration published yet?
v1.01, 28 January 2021¶
first public version