Speedup Retrievals with Striping (Mistral only)

file version: 11 November 2021

current slk version: 3.3.16

Overview

On a lustre file system, the content of files is served from so called object storage targets (OSTs). Lustre allows to configure per file or per folder onto how many OSTs a file’s content is split. This configuration option is denoted as striping. striping=1 means that the file’s content is served by on OST only. striping=4 (for example) means that the file’s content is split into four distinct parts which are served by four OSTs – one part per OST. On the lustre file system, which is connected to Mistral, all files are by default stored with striping set to 1. The default configuration on the lustre file system of Levante might differ. The general advantages and disadvantanges of striping are not described here. There are numerous online sources on this topic.

The retrieval speed of files larger than approximately 50 GB is enhanced when striping is set to a value > 1 for the retrieved file. This has to be done prior to file retrieval. For files of 200 GB size we suggest to set striping to 8 to obtain the highest possible transfer rate. The whole lustre file system might be slowed down if the current lustre file system is generally busy and many large files are retrieved with striping set to 1, the whole lustre file system might be slowed down. Therefore, it is favourable for the whole system performance that files above 50 GB size are retrieved with striping >1.

Set and Check Striping

Striping has to be set prior to file creation via this command:

lfs setstripe -S 4M -c 8 TARGET

TARGET can either be the soon-to-be created/retrieved file or a directory in/into which files are created/retrieved. The command cannot be applied on files that exist already. If the command is applied on a directory that contains files then the old files’ striping remains unchanged but new files are striped as defined.

The striping of a file can be checked via:

lfs getstripe -d TARGET

Example

We have an archived file /arch/ab01234/c567890/test01.nc and create three local directories retr01, retr02 and retr03. The example file only has a size of 105MB. In real applications, the file should have a size of approximately at least 50 GB.

No Striping

We will retrieve the archived file into retr01 without setting any striping and check the striping afterwards.

$ mkdir retr01

$ slk retrieve /arch/ab01234/c567890/test01.nc retr01
[========================================|] 100% complete. Files retrieved: 1/1, [105.5M/105.5M].

$ lfs getstripe -d retr_01/test01.nc
lmm_stripe_count:  1
lmm_stripe_size:   1048576
lmm_pattern:       raid0
lmm_layout_gen:    0
lmm_stripe_offset: 34
    obdidx       objid        objid    group
        34    25319089    0x18256b1        0

Set Striping for a file

We will retrieve the archived file into retr02 and set striping to 8 prior to retrieval. Afterwards, we check the striping.

$ mkdir retr02

$ lfs setstripe -S 4M -c 8 retr_02/test01.nc

$ slk retrieve /arch/ab01234/c567890/test01.nc retr02
[========================================|] 100% complete. Files retrieved: 1/1, [105.5M/105.5M].

$ lfs getstripe -d retr_02/test01.nc
lmm_stripe_count:  8
lmm_stripe_size:   4194304
lmm_pattern:       raid0
lmm_layout_gen:    0
lmm_stripe_offset: 95
    obdidx       objid        objid    group
        95    25298851    0x18207a3        0
        62    25301879    0x1821377        0
        82    27774549    0x1a7ce55        0
         2    28747311    0x1b6a62f        0
        79    25442734    0x18439ae        0
        15    28451203    0x1b22183        0
       118    20014121    0x1316429        0
       119    19860330    0x12f0b6a        0

Set Striping for a directory

First, we create a file test02.txt in retr03. Second, we set striping of the directory retr03 to 8. Finally, we retrieve the file and check the striping.

$ mkdir retr03

$ echo "abc" > retr03/test02.txt

$ lfs setstripe -S 4M -c 8 retr_03

$ slk retrieve /arch/ab01234/c567890/test01.nc retr03
[========================================|] 100% complete. Files retrieved: 1/1, [105.5M/105.5M].

$ lfs getstripe -d retr03/test01.nc
lmm_stripe_count:  8
lmm_stripe_size:   4194304
lmm_pattern:       raid0
lmm_layout_gen:    0
lmm_stripe_offset: 80
    obdidx       objid        objid    group
        80    27491641    0x1a37d39        0
        31    25444333    0x1843fed        0
        64    25053678    0x17e49ee        0
        61    25355939    0x182e6a3        0
         6    28884651    0x1b8beab        0
       111    24873595    0x17b8a7b        0
        79    25442733    0x18439ad        0
        73    25332380    0x1828a9c        0

$ lfs getstripe -d retr03/test02.txt
lmm_stripe_count:  1
lmm_stripe_size:   1048576
lmm_pattern:       raid0
lmm_layout_gen:    0
lmm_stripe_offset: 26
    obdidx       objid        objid    group
        26    25574671    0x1863d0f        0