(un)packems#

file version: 5 Oct 2026

The packems package was originally designed to simplify the storage of small files on tape. In the past, small files were not automatically packed by our software prior to being written to tape. Therefore, users had to pack their data manually before their archival. The original version of packems packed sets of small files into tar balls of a defined size and archived them. The information on which files are stored in which tar ball is stored in INDEX.txt files archived next to the tar balls. Tools to list an retrieved data which were archived by packems were also created. In total, the packems package consisted of these tools:

  • packems: pack & archive data

  • listems: list archived packed data

  • unpackems: retrieved packed data from tape and unpack them

The new Versity HSM software packs small files automatically. However, we still recommend to pack large sets of small files as described in Storage options, quota and file size.

packems#

The archival feature of packems has been removed. It can be still used for packing data.

What packems does:

  • create a Makefile (*.mk) which is run later for the actual packing

  • create lists of files to be packed per tar ball => *.lst files

  • run Makefile: create tars, write content to *.idx files and merge to INDEX.txt

packems becomes available by loading the module packems:

module load packems

Usage examples are given below.

pack a folder#

# check out what's there
$ ls -la
drwxrwxr-x+  4 k204221 ka1209         4096 Oct  6 17:08 .
drwxrwxr-x+ 39 k204221 ka1209         4096 Oct  6 01:45 ..
drwxr-xr-x+  4 k204221 ka1209         4096 Oct  6 01:54 my-data

# create folder for packs
$ mkdir packed_a

# create packs
# -S destination in tape archive: no meaning because we pack only
# -o base name of tar balls
# -d local storage location of tar balls
# default size of tar balls (nothing set)
$ packems my-data -S /hsm/arch/bm0146/k204221 -o tar-ball-base-name -d packed_a
...

# check packing output
$ ls -la packed_a
drwxrwxr-x+  4 k204221 ka1209         4096 Oct  6 17:08 .
drwxrwxr-x+ 39 k204221 ka1209         4096 Oct  6 01:45 ..
-rw-rw-r--   1 k204221 ka1209 100813721283 Oct  6 17:07 tar-ball-base-name_001.tar
-rw-rw-r--   1 k204221 ka1209      4723232 Oct  6 17:07 tar-ball-base-name_001.tar.idx
-rw-rw-r--   1 k204221 ka1209      3787061 Oct  6 16:47 tar-ball-base-name_001.tar.lst
-rw-rw-r--   1 k204221 ka1209  30872721123 Oct  6 17:07 tar-ball-base-name_002.tar
-rw-rw-r--   1 k204221 ka1209      2433271 Oct  6 17:07 tar-ball-base-name_002.tar.idx
-rw-rw-r--   1 k204221 ka1209      2380022 Oct  6 16:47 tar-ball-base-name_002.tar.lst
-rw-rw-r--   1 k204221 ka1209      6291289 Oct  6 17:08 INDEX.txt
-rw-rw-r--   1 k204221 ka1209      6382301 Oct  6 16:47 tar-ball-base-name.mk

# copy packs to archive
$ module load tape
$ atlas cp `pwd`/packed_a/*.tar `pwd`/packed_a/INDEX.txt /hsm/arch/bm0146/k204221/data_a/

pack a folder; change tar ball size#

# check out what's there
$ ls -la
drwxrwxr-x+  4 k204221 ka1209         4096 Oct  6 17:08 .
drwxrwxr-x+ 39 k204221 ka1209         4096 Oct  6 01:45 ..
drwxr-xr-x+  4 k204221 ka1209         4096 Oct  6 01:54 my-data

# create folder for packs
$ mkdir packed_b

# create packs
# -S: destination in tape archive: no meaning because we pack only
# -o: base name of tar balls
# -d: local storage location of tar balls
# -m: set max size of tar balls from 100 GB (default) to 200 GB
# -t: set target size of tar balls to 190 GB
$ packems my-data -S /hsm/arch/bm0146/k204221 -o tar-ball-base-name -d packed_b -m 200 -t 190
...

# check packing output
$ ls -la packed_b
drwxrwxr-x+  4 k204221 ka1209         4096 Oct  6 17:08 .
drwxrwxr-x+ 39 k204221 ka1209         4096 Oct  6 01:45 ..
-rw-rw-r--   1 k204221 ka1209 130871721984 Oct  6 17:07 tar-ball-base-name_001.tar
-rw-rw-r--   1 k204221 ka1209      5733273 Oct  6 17:07 tar-ball-base-name_001.tar.idx
-rw-rw-r--   1 k204221 ka1209      4387071 Oct  6 16:47 tar-ball-base-name_001.tar.lst
-rw-rw-r--   1 k204221 ka1209      6291289 Oct  6 17:08 INDEX.txt
-rw-rw-r--   1 k204221 ka1209      6382301 Oct  6 16:47 tar-ball-base-name.mk

# copy packs to archive
$ module load tape
$ atlas cp `pwd`/packed_b/*.tar `pwd`/packed_b/INDEX.txt /hsm/arch/bm0146/k204221/data_b/

pack all netCDF files#

# check out what's there
$ ls -la
drwxrwxr-x+  4 k204221 ka1209         4096 Oct  6 17:08 .
drwxrwxr-x+ 39 k204221 ka1209         4096 Oct  6 01:45 ..
drwxr-xr-x+  4 k204221 ka1209         4096 Oct  6 01:54 my-data

# create folder for packs
$ mkdir packed_c

# create packs
# -i -: read from stdin (=> we pipe into packems)
# -S: destination in tape archive: no meaning because we pack only
# -o: base name of tar balls
# -d: local storage location of tar balls
$ find my-data -type f -name "*.nc" | packems -i - -S /hsm/arch/bm0146/k204221 -o only-netcdf-files-packed -d packed_c
...

# check packing output
$ ls -la packed_c
drwxrwxr-x+  4 k204221 ka1209         4096 Oct  6 17:08 .
drwxrwxr-x+ 39 k204221 ka1209         4096 Oct  6 01:45 ..
-rw-rw-r--   1 k204221 ka1209  93212425341 Oct  6 17:07 only-netcdf-files-packed_001.tar
-rw-rw-r--   1 k204221 ka1209      2523211 Oct  6 17:07 only-netcdf-files-packed_001.tar.idx
-rw-rw-r--   1 k204221 ka1209      2412432 Oct  6 16:47 only-netcdf-files-packed_001.tar.lst
-rw-rw-r--   1 k204221 ka1209      3252321 Oct  6 17:08 INDEX.txt
-rw-rw-r--   1 k204221 ka1209      1242135 Oct  6 16:47 only-netcdf-files-packed.mk

# copy packs to archive
$ module load tape
$ atlas cp `pwd`/packed_c/*.tar `pwd`/packed_b/INDEX.txt /hsm/arch/bm0146/k204221/data_c/

Details on packems are given here: https://code.mpimet.mpg.de/projects/esmenv/wiki/Packems

listems#

Not adapted to new HSM yet. Please search through your INDEX.txt files manually.

unpackems#

Not adapted to new HSM yet. Please search through your INDEX.txt files manually and get the tar balls with atlas cp.

On this page