Speedup Retrievals with Striping¶
file version: 20 April 2022
current slk version: 3.3.21
Overview¶
Note
On mistral, striping is mandatory for retrieval of files > 10 GB to sustain high lustre I/O performance for all users. We recommend to use striping on levante for the time being. This recommendation will change at some point in future.
On a lustre file system, the content of files is served from so called object storage targets (OSTs). Lustre allows to configure per file or per folder onto how many OSTs a file’s content is split. This configuration option is denoted as striping. striping=1 means that the file’s content is served by on OST only. striping=4 (for example) means that the file’s content is split into four distinct parts which are served by four OSTs – one part per OST. On the lustre file system, which is connected to Mistral, all files are by default stored with striping set to 1. The default configuration on the lustre file system of levante has still to be decided. The general advantages and disadvantages of striping are not described here. There are numerous online sources on this topic.
On mistral: Striping is mandatory. The lustre file system will probably slow down or start hanging when several large files (> 10 GB) are retrieved via slk with striping set to 1. This will affect many other users working on mistral at the same time. Therefore, we strongly urge you to use striping for files > 10 GB. We suggest to set striping to 8. It might be even reasonable to use striping for small files when many files are retrieved at once. The striping factor has to be set prior to file retrieval. It has to be set on folder level and not on file level. Setting it on file level does not work because files get a temporary non-predictable name during the retrieval and are renamed after successful retrieval.
On levante: Striping is recommended. We do not expect that the lustre file system of levante will slow down as the lustre file system of mistral. Nevertheless, we recommend using striping for the time being. When you set up striping, it is highly recommended not to use a constant striping factor but set up progressive file layout (= automatic striping based on file size).
Note
The lustre file system of mistral does not support progressive file layout. Therefore, a constant striping factor has to be used on mistral.
Set and Check Striping¶
Striping has to be set prior to file creation via one of these two commands:
# ON MISTRAL (constant striping factor)
lfs setstripe -S 4M -c 8 TARGET
# all files have striping factor of 8
# ON LEVANTE (progressive file layout)
lfs setstripe -E 1G -c 1 -S 1M -E 4G -c 4 -S 1M -E -1 -c 8 -S 1M TARGET
# file size < 1 GB => striping factor of 1
# file size < 4 GB => striping factor OF 4
# files size >= 4 GB;
TARGET can be either the soon-to-be created/retrieved file or a directory in/into which files are created/retrieved. The command cannot be applied on files that exist already. If the command is applied on a directory that contains files then the old files’ striping remains unchanged but new files are striped as defined.
The striping of a file can be checked via:
lfs getstripe -d TARGET
Example¶
We have an archived file /arch/ab01234/c567890/test01.nc and create three local directories retr01, retr02 and retr03. The example file only has a size of 105MB. In real applications, the file should have a size of approximately at least 50 GB.
No Striping¶
We will retrieve the archived file into retr01 without setting any striping and check the striping afterwards.
$ mkdir retr01
$ slk retrieve /arch/ab01234/c567890/test01.nc retr01
[========================================|] 100% complete. Files retrieved: 1/1, [105.5M/105.5M].
$ lfs getstripe -d retr_01/test01.nc
lmm_stripe_count: 1
lmm_stripe_size: 1048576
lmm_pattern: raid0
lmm_layout_gen: 0
lmm_stripe_offset: 34
obdidx objid objid group
34 25319089 0x18256b1 0
Set Striping for a file¶
We will retrieve the archived file into retr02 and set striping to 8 prior to retrieval. Afterwards, we check the striping.
$ mkdir retr02
$ lfs setstripe -S 4M -c 8 retr_02/test01.nc
$ slk retrieve /arch/ab01234/c567890/test01.nc retr02
[========================================|] 100% complete. Files retrieved: 1/1, [105.5M/105.5M].
$ lfs getstripe -d retr_02/test01.nc
lmm_stripe_count: 8
lmm_stripe_size: 4194304
lmm_pattern: raid0
lmm_layout_gen: 0
lmm_stripe_offset: 95
obdidx objid objid group
95 25298851 0x18207a3 0
62 25301879 0x1821377 0
82 27774549 0x1a7ce55 0
2 28747311 0x1b6a62f 0
79 25442734 0x18439ae 0
15 28451203 0x1b22183 0
118 20014121 0x1316429 0
119 19860330 0x12f0b6a 0
Set Striping with a constant striping factor for a directory¶
First, we create a file test02.txt in retr03. Second, we set striping of the directory retr03 to 8. Finally, we retrieve the file and check the striping.
$ mkdir retr03
$ echo "abc" > retr03/test02.txt
$ lfs setstripe -S 4M -c 8 retr_03
$ slk retrieve /arch/ab01234/c567890/test01.nc retr03
[========================================|] 100% complete. Files retrieved: 1/1, [105.5M/105.5M].
$ lfs getstripe -d retr03/test01.nc
lmm_stripe_count: 8
lmm_stripe_size: 4194304
lmm_pattern: raid0
lmm_layout_gen: 0
lmm_stripe_offset: 80
obdidx objid objid group
80 27491641 0x1a37d39 0
31 25444333 0x1843fed 0
64 25053678 0x17e49ee 0
61 25355939 0x182e6a3 0
6 28884651 0x1b8beab 0
111 24873595 0x17b8a7b 0
79 25442733 0x18439ad 0
73 25332380 0x1828a9c 0
$ lfs getstripe -d retr03/test02.txt
lmm_stripe_count: 1
lmm_stripe_size: 1048576
lmm_pattern: raid0
lmm_layout_gen: 0
lmm_stripe_offset: 26
obdidx objid objid group
26 25574671 0x1863d0f 0
Set Striping with progressive file layout for a directory¶
First, we create a file test03.txt in retr04. Second, we set striping of the directory retr04 to a progressive file layout. Finally, we retrieve the file and check the striping.
$ mkdir retr04
$ echo "abc" > retr04/test03.txt
$ lfs setstripe -E 1G -c 1 -S 1M -E 4G -c 4 -S 1M -E -1 -c 8 -S 1M retr_04
$ slk retrieve /arch/ab01234/c567890/test01.nc retr04
[========================================|] 100% complete. Files retrieved: 1/1, [105.5M/105.5M].
$ lfs getstripe -d retr03/test01.nc
lcm_layout_gen: 0
lcm_mirror_count: 1
lcm_entry_count: 3
lcme_id: N/A
lcme_mirror_id: N/A
lcme_flags: 0
lcme_extent.e_start: 0
lcme_extent.e_end: 1073741824
stripe_count: 1 stripe_size: 1048576 pattern: raid0 stripe_offset: -1
lcme_id: N/A
lcme_mirror_id: N/A
lcme_flags: 0
lcme_extent.e_start: 1073741824
lcme_extent.e_end: 4294967296
stripe_count: 4 stripe_size: 1048576 pattern: raid0 stripe_offset: -1
lcme_id: N/A
lcme_mirror_id: N/A
lcme_flags: 0
lcme_extent.e_start: 4294967296
lcme_extent.e_end: EOF
stripe_count: 8 stripe_size: 1048576 pattern: raid0 stripe_offset: -1
$ lfs getstripe -d retr04/test03.txt
lmm_stripe_count: 1
lmm_stripe_size: 1048576
lmm_pattern: raid0
lmm_layout_gen: 0
lmm_stripe_offset: 26
obdidx objid objid group
26 25574671 0x1863d0f 0