Speedup Retrievals with Striping (Mistral only)¶
file version: 11 November 2021
current slk version: 3.3.16
On a lustre file system, the content of files is served from so called object storage targets (OSTs). Lustre allows to configure per file or per folder onto how many OSTs a file’s content is split. This configuration option is denoted as
striping=1 means that the file’s content is served by on OST only.
striping=4 (for example) means that the file’s content is split into four distinct parts which are served by four OSTs – one part per OST. On the lustre file system, which is connected to Mistral, all files are by default stored with
striping set to
1. The default configuration on the lustre file system of Levante might differ. The general advantages and disadvantanges of striping are not described here. There are numerous online sources on this topic.
The retrieval speed of files larger than approximately 50 GB is enhanced when striping is set to a value
> 1 for the retrieved file. This has to be done prior to file retrieval. For files of 200 GB size we suggest to set striping to
8 to obtain the highest possible transfer rate. The whole lustre file system might be slowed down if the current lustre file system is generally busy and many large files are retrieved with striping set to
1, the whole lustre file system might be slowed down. Therefore, it is favourable for the whole system performance that files above 50 GB size are retrieved with striping
Set and Check Striping¶
Striping has to be set prior to file creation via this command:
lfs setstripe -S 4M -c 8 TARGET
TARGET can either be the soon-to-be created/retrieved file or a directory in/into which files are created/retrieved. The command cannot be applied on files that exist already. If the command is applied on a directory that contains files then the old files’ striping remains unchanged but new files are striped as defined.
The striping of a file can be checked via:
lfs getstripe -d TARGET
We have an archived file
/arch/ab01234/c567890/test01.nc and create three local directories
retr03. The example file only has a size of 105MB. In real applications, the file should have a size of approximately at least 50 GB.
We will retrieve the archived file into
retr01 without setting any striping and check the striping afterwards.
$ mkdir retr01 $ slk retrieve /arch/ab01234/c567890/test01.nc retr01 [========================================|] 100% complete. Files retrieved: 1/1, [105.5M/105.5M]. $ lfs getstripe -d retr_01/test01.nc lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 34 obdidx objid objid group 34 25319089 0x18256b1 0
Set Striping for a file¶
We will retrieve the archived file into
retr02 and set striping to
8 prior to retrieval. Afterwards, we check the striping.
$ mkdir retr02 $ lfs setstripe -S 4M -c 8 retr_02/test01.nc $ slk retrieve /arch/ab01234/c567890/test01.nc retr02 [========================================|] 100% complete. Files retrieved: 1/1, [105.5M/105.5M]. $ lfs getstripe -d retr_02/test01.nc lmm_stripe_count: 8 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 95 obdidx objid objid group 95 25298851 0x18207a3 0 62 25301879 0x1821377 0 82 27774549 0x1a7ce55 0 2 28747311 0x1b6a62f 0 79 25442734 0x18439ae 0 15 28451203 0x1b22183 0 118 20014121 0x1316429 0 119 19860330 0x12f0b6a 0
Set Striping for a directory¶
First, we create a file
retr03. Second, we set striping of the directory
8. Finally, we retrieve the file and check the striping.
$ mkdir retr03 $ echo "abc" > retr03/test02.txt $ lfs setstripe -S 4M -c 8 retr_03 $ slk retrieve /arch/ab01234/c567890/test01.nc retr03 [========================================|] 100% complete. Files retrieved: 1/1, [105.5M/105.5M]. $ lfs getstripe -d retr03/test01.nc lmm_stripe_count: 8 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 80 obdidx objid objid group 80 27491641 0x1a37d39 0 31 25444333 0x1843fed 0 64 25053678 0x17e49ee 0 61 25355939 0x182e6a3 0 6 28884651 0x1b8beab 0 111 24873595 0x17b8a7b 0 79 25442733 0x18439ad 0 73 25332380 0x1828a9c 0 $ lfs getstripe -d retr03/test02.txt lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 26 obdidx objid objid group 26 25574671 0x1863d0f 0