HSM News Vol 1

This first HSM news blog entry is about usage hints / recommendations for the next HSM system by Versity.

TOC

• Symbolic links: ignored

• No write perssions on /work

• Use absolute paths

• Combine multiple staging and copy requests

• Append Slash to destination paths

• Retrieval: combining stage and cp

• How to see whether a file is cached?

• Filter job list

Symbolic links: ignored

atlas cp transfers only folders and regular files. Symbolic links are ignored without any warning. This might change in future releases and will be announced when this is the case.

No write perssions on /work

Currently, atlas is not allowed to write into /work but only into /scratch Therefore, retrievals can only target /scratch. We are aware that this is no ideal solution. But we do not want to permit a new software to have full write permissions on /work in order to ensure data integrity. We cannot provide a timeline when atlas receives write access.

Use absolute paths

When used as command on the terminal or in scripts, atlas requires absolutes paths and is not aware of the current working directory.

Thus, this does not work:

$ # ~~~~~~~~~~~ DOES NOT WORK! ~~~~~~~~~~~
$ cd /scratch/k/k204221/test_data
$ ls
test.txt
$ atlas cp test.txt /hsm/arch/bm0146/k204221/my_data/
$ # ~~~~~~~~~~~ DOES NOT WORK! ~~~~~~~~~~~

Instead, the absolute path of test.txt has to be used:

$ cd /scratch/k/k204221/test_data
$ ls
test.txt
$ atlas cp /scratch/k/k204221/test_data/test.txt /hsm/arch/bm0146/k204221/my_data/

However, when you use the atlas shell, relative paths work:

$ atlas
# we enter the atlas shell
> cd /scratch/k/k204221/test_data
# use commands without prefix "atlas " here:
> cp test.txt /hsm/arch/bm0146/k204221/my_data/

Combine multiple staging and copy requests

We strongly recommend to combine multiple small staging requests into one large request – and similarly for copy jobs. You’ll have fewer jobs running in ATLAS which results in a better overview and simplifies debugging. For the processes within Versity, it does not matter whether 100 staging requests for one file each or one staging request for 100 files come in. However, please send all staging requests within a short timespan so that Versity knows all required files and can optimize the tape access.

# we have some files in the archive:
$ atlas ls /hsm/arch/bm0146/k204221/iow | sort | head -n 7
INDEX.txt
iow_data_001.tar
iow_data_002.tar
iow_data_003.tar
iow_data_004.tar
iow_data_005.tar
iow_data_006.tar

# ~~~~~~~~~~~~~~~~ NOT RECOMMENDED ~~~~~~~~~~~~~~~~
# this works fine in Versity and with the Versity tape backend
$ atlas stage /hsm/arch/bm0146/k204221/iow/INDEX.txt
stage job submitted: 16746
$ atlas stage /hsm/arch/bm0146/k204221/iow/iow_data_001.tar
stage job submitted: 16747
$ atlas stage /hsm/arch/bm0146/k204221/iow/iow_data_002.tar
stage job submitted: 16748
$ atlas stage /hsm/arch/bm0146/k204221/iow/iow_data_003.tar
stage job submitted: 16749
$ atlas stage /hsm/arch/bm0146/k204221/iow/iow_data_004.tar
stage job submitted: 16750
$ atlas stage /hsm/arch/bm0146/k204221/iow/iow_data_005.tar
stage job submitted: 16751
$ atlas stage /hsm/arch/bm0146/k204221/iow/iow_data_006.tar
stage job submitted: 16752
# however, this creates 7 jobs to monitor -- or to check for failure

# ~~~~~~~~~~~~~~~~~~ RECOMMENDED ~~~~~~~~~~~~~~~~~~
# from a usage perspective we propose this:
$ atlas stage /hsm/arch/bm0146/k204221/iow/INDEX.txt '/hsm/arch/bm0146/k204221/iow/iow_data_00*.tar'
stage job submitted: 16753

# similar for the copy process, we would suggest to do this:
$ atlas cp /hsm/arch/bm0146/k204221/iow/INDEX.txt '/hsm/arch/bm0146/k204221/iow/iow_data_00*.tar' /scratch/k/k204221/my_data/

# alternatively, you can use `-t <destination>`
$ atlas cp /hsm/arch/bm0146/k204221/iow/INDEX.txt '/hsm/arch/bm0146/k204221/iow/iow_data_00*.tar' -t /scratch/k/k204221/my_data/

Note

Please note that wildcards in paths of the tape archive are expanded by atlas itself and not by your shell. Therefore, please put such paths into single quotes so that your shell passes the pattern on unchanged.

Note

Please always append / to the destination path.

Append slash to destination paths

Please always append / to destination paths in cp.

# destination folder for test archivals
$ atlas ls /hsm/arch/bm0146/k204221/testing
# empty!

# We will run three 'cp' commands with slight variations and show the importance
# of appending '/' to destination paths.

# copy to non-existing destination path without '/' in the end
$ atlas cp /scratch/k/k204221/testdata/test.txt /hsm/arch/bm0146/k204221/testing/caseA/a
copy job submitted: 20123

# copy to non-existing destination path with '/' in the end
$ atlas cp /scratch/k/k204221/testdata/test.txt /hsm/arch/bm0146/k204221/testing/caseB/b/
copy job submitted: 20124

# create destination path manually and, then, transfer into existing path without '/' in the end
$ atlas mkdir -p /hsm/arch/bm0146/k204221/testing/caseC/c
$ atlas cp /scratch/k/k204221/testdata/test.txt /hsm/arch/bm0146/k204221/testing/caseC/c
copy job submitted: 20125

# archivals done:
$ atlas ls /hsm/arch/bm0146/k204221/testing
caseA
caseB
caseC

# Now, let's check what is in the folders caseA to caseC
#  a file with name 'a'!
$ atlas ls -la /hsm/arch/bm0146/k204221/testing/caseA
-rw-r--r--  k204221  bm0146  4B  Jul 24 22:00  -o--v    a
# a folder with name 'b'
$ atlas ls -la /hsm/arch/bm0146/k204221/testing/caseB
drwxr-xr-x  k204221  bm0146  8B  Jul 24 22:02  -----    b
# a folder with name 'c'
$ atlas ls -la /hsm/arch/bm0146/k204221/testing/caseC
drwxr-xr-x  k204221  bm0146  8B  Jul 24 22:03  -----    c

Retrieval: combining stage and cp

When you want to transfer data from /hsm to somewhere else, all four transfer commands can only transfer cached files (online) and do not automatically stage files from tape. This will change in ATLAS version 4 which is to be released in end of August 2026. Until then, please run atlas stage manually to get the files back into the cache. The command will submit a stage job which status can be checked via atlas jobs get. You can run the transfer command of your choice while the staging is running. However, only those file, which were cached when the transfer command was started, will be transfered. Therefore, the transfer command has to be run at least once after the stage job ended. We provide a script template further below where you just need to update src and dst. This is only required until the release of ATLAS version 4 in the end of August.

Currently, atlas cp /hsm/... (/work|/scratch)/... can only transfer cached files and does not automatically stage files from tape. This will change in ATLAS version 4 which is to be released in end of August 2026. Until then, atlas stage has to be run manually to get the files back into the cache. The stage command will submit a stage job which status can be checked via atlas jobs get. You can run atlas cp already while the staging job is still running. However, only those file, which were cached when the transfer command was started, will be transfered. Therefore, the transfer command has to be run at least once after the stage job ended.

Below, please find a template for file retrieval combining atlas stage and atlas cp. The copy process is started already while the staging is still ongoing. As soon as ATLAS version 4 will be release in the end of August 2026, atlas cp will automatically stage files and this template becomes unnecessary.

# TODO: adapt these two lines!
src=/hsm/arch/bm0146/k204221/my_data
dst=/scratch/k/k204221/blub/
#
stage_output=`atlas stage --recursive ${src}`
echo ${stage_output}
stage_job_id=`echo ${stage_output} | sed -E 's/^[^0-9]*([0-9]+)[^0-9]*$/\1/g'`

# some initial values
stage_job_state=running
copy_job_state=none
copy_job_id=0
# As long as the staging job is running, we submit copy jobs; when the staging job ended we will start one final iteration of copying
final_copy_job_submitted=false
while [ "$stage_job_state" = "running" ] || [ "$copy_job_state" = "running" ] || [ "${final_copy_job_submitted}" != "true" ]; do
    sleep 300
    if [ "$copy_job_state" = "running" ]; then
        # copy job is still running
        echo "previous copy job ${copy_job_id} still running"
    else
        # copy job not running anymore => restart copy job
        # '--update' makes the copy job skip files which were already
        # retrieved by one of the previous copy jobs
        copy_output=`atlas cp --recursive --update ${src} ${dst}/`
        echo ${copy_output}
        copy_job_id=`echo ${copy_output} | sed -E 's/^[^0-9]*([0-9]+)[^0-9]*$/\1/g'`
        if [ "$stage_job_state" != "running" ]; then
            # When we arrive here, the stage job has ended and a new copy job
            # has been submitted. In this situation we would like to wait until
            # this final copy job is finished and exit the loop afterwards. This
            # is controlled by 'final_copy_job_submitted'
            final_copy_job_submitted="true"
        fi
        # give the new copy job some time to initialize
        sleep 10
    fi

    stage_job_raw_state=`atlas jobs get ${stage_job_id} | grep "State:"`
    stage_job_state=`echo ${stage_job_raw_state} | sed 's/^State:[ ]*//g'`
    echo "stage job ${stage_job_id} ${stage_job_raw_state}"
    copy_job_raw_state=`atlas jobs get ${copy_job_id} | grep "State:" | sed 's/^State:[ ]*//g'`
    copy_job_state=`echo ${copy_job_raw_state} | sed 's/^State:[ ]*//g'`
    echo "copy job ${copy_job_id} ${copy_job_raw_state}"
    echo "current stage job state is '${stage_job_state}'"
    echo "current copy job state is '${copy_job_state}'"
done

# The intermediate copy jobs are expected to report errors for files which
# were not staged yet. Therefore, please check the last copy job -- it is the
# one which ran after the staging had been finished:
atlas jobs get ${copy_job_id}

How to see whether a file is cached?

How to see whether a file is cached? Is there a isCache command? No, but atlas ls -la … provides you with all information you need! Example:

$ atlas ls /hsm/arch/bm0146/k204221/iow -la
-rwxr-xrwx  k204221  ka1209  1.2M    Oct 16 20:43  an--v  3  INDEX.txt
-rwxr-xr-x  k204221  ka1209  19.5G   Dec 05 10:11  ao---  2  iow_data_001.tar
-rwxr-xr-x  k204221  ka1209  19.0G   Dec 02 13:10  ao---  2  iow_data_002.tar
-rwxr-xr-x  k204221  ka1209  19.5G   Dec 05 10:11  a----  2  iow_data2_001.tar
-rwxr-xr-x  k204221  ka1209  19.0G   Dec 05 10:15  a----  2  iow_data2_002.tar
-rwxr-xr-x  k204221  ka1209  186.9G  Dec 05 10:25  a-s--  2  iow_data3_001.tar
-rwxr-xr-x  k204221  ka1209  24.6G   Dec 05 10:14  aw---  2  iow_data3_002.tar
-rw-r--r--  k204221  bm0146  4.3B    Jul 24 22:00  -o--v  2  new_data_001.tar

The third column from right contains a lot of useful information. The second character indicates the cache state:

Cache flags:
  -o---    File is online
  -p---    File is partially online (mixed online/offline blocks)
  -n---    File is marked as release never (== online permanently)
  -w---    File is offline but has a warm-tier copy (intermediate HDD storage)
  -----    File is offline

Applying this to the atlas ls output from above means:

• cached files: INDEX.txt, iow_data_*.tar and new_data_001.tar

• files only on tape (staging required): iow_data2_*.tar

• file in an intermediate storage (staging required; quick): iow_data3_002

• file currently bein staged (see below for details): iow_data3_001

Meaning of other characters at other positions:

Archive flags:
  a----    File is completely archived
  n----    File is marked as no archive
  -----    File is not archived

Staging flags:
  --s--    File is staging
  --X--    File is marked as stage failed
  -----    File is not staging

Retention flags:
  ---r-    File has retention set
  -----    File does not have retention set

Checksum flags:
  ----n    File has checksum set but has not been verified
  ----v    File has checksum set and has been verified
  ----X    File has checksum set but verified invalid
  -----    File does not have a user checksum

Filter job list

The command atlas jobs prints all of your copy, move and staging jobs of the previous week. These can be a very large number of jobs. The command atlas jobs list allows to filter the output. Here a few examples:

# print your 50 recent running staging jobs
#  (printing 50 results is the default)
atlas jobs list --state running --type stage

# print your last 10 failed jobs:
atlas jobs list --state failed --limit 10

For details please run atlas jobs list --help.

Container mode now available in the advanced spawner