Fast File System find and disk usage Tools#

The basic unix commands du and find can take minutes to hours on these file systems. As this is not practicable for large projects we provide the two fast alternatives limscaup-find and limscaup-du taking a few seconds to a few minutes for large searches. For reasons of basic data privacy, your searches are limited to your own projects and files.

While luv already points out which fraction of files are unused for quite some time, these tools come in handy to find out where these are actually located.

Note

These tools query a dedicated metadata lake in the background. The metadata lake gets updated nightly. If you have created, modified, or deleted files during the day, those changes will not appear until after the next update. Due to maintenance and other issues, the metadata lake might not be updated, please check the last_db_update column to check the last update time of the metadata lake. This can vary between mounts, e.g. /work and /shared might be updated at different times, while minio-s3 is updated more frequently.

All interaction is done via the command-line commands described below, direct access to the metadata lake is neither required nor recommended.

Available Tools#

The following commands are available:

Command	Purpose
`limscaup-find`	Search for files and paths matching a pattern
`limscaup-du`	Compute directory usage (sizes, file counts, etc.)

Each command supports --help for an overview of usage and options.

Common Features#

Works on the following mounts:
- /work
- /home
- /shared
- /minio-s3
Wildcards follow the standard shell (GLOB) syntax:
- * matches any sequence of characters
- ? matches a single character
Output can be saved as .parquet, .csv or .json.
Filtering supports DuckDB SQL predicates, e.g. "size > 1e6 AND atime_ms < '2025-11-01'" (needs both quotes).

The following fields can be added via --add or used in --filter:

Field	Mounts	Description
`size`	work, home, shared, minio-s3	File size in bytes
`ctime_ms`	work, home, shared, minio-s3	Change time (inode metadata change), in milliseconds, accepts dates as quoted ‘YYYY-MM-DD’ strings
`mtime_ms`	work, home, shared, minio-s3	Modification time (content change), in milliseconds
`atime_ms`	work, home, shared, minio-s3	Last access time, in milliseconds
`crtime_ms`	work, home, shared, minio-s3	Creation time, in milliseconds
`path`	work, home, shared, minio-s3	Full absolute path
`ino`	work, home, shared, minio-s3	Inode number (unique per filesystem)
`uid`	work, home, shared, minio-s3	User ID of the file owner
`gid`	work, home, shared, minio-s3	Group ID of the file owner
`projid`	work, home, shared, minio-s3	Project ID used for project-based quotas
`mode`	work, home, shared, minio-s3	File type and permissions (POSIX `st_mode` bitmask; ACL flags like `+` in `ls -l` are not included)
`nlink`	home, shared, minio-s3	Number of hard links to the file
`owner_sid`	home, shared, minio-s3	Owner SID (Windows ACL)
`group_sid`	home, shared, minio-s3	Group SID (Windows ACL)
`cluster`	minio-s3	Cluster name (e.g. `c1` or `c2`)
`bucket`	minio-s3	Bucket name
`key`	minio-s3	Object key (path within the bucket)
`exists`	minio-s3	Whether the object exists, tombstones are deleted at utc midnight
`etag`	minio-s3	ETag of the object (MD5 checksum for non-multipart objects)
`headtime_ms`	minio-s3	Last time the object was HEADed, in milliseconds
`deltime_ms`	minio-s3	Last time the object was deleted, in milliseconds
`daily_access_count`	minio-s3	Number of times the object was accessed this day, gets reset at utc midnight
`daily_head_count`	minio-s3	Number of times the object was HEADed this day, gets reset at utc midnight
`total_get_count`	minio-s3	Total number of times the object was GETed
`total_head_count`	minio-s3	Total number of times the object was HEADed

limscaup-find#

limscaup-find locates paths or files matching a given pattern. It behaves similar to find on Linux but is backed by the LimScaup metadata lake, which is significantly faster for large directories.

Examples

Warning

When using wildcards (*, ?), quote the pattern to avoid shell interpretation.

You can only search within your own projects and files.

# Find all .nc files in a project
limscaup-find "/work/ik1017/CMIP6/data/CMIP6/*.nc" # adjust to one of your projects

# Add extra columns to the output (e.g. uid, gid)
limscaup-find "/work/ik1017/*.nc" --add uid --add gid

# Filter by file size and access time using SQL. Note the quotes!
limscaup-find "/work/ik1017/*.nc" \
    --filter "size > 5e6 AND atime_ms >= '2025-06-15'"

# Change the number of rows printed
limscaup-find "/work/ik1017/*.nc" --max_rows 50

# Save result to a parquet file
limscaup-find "/work/ik1017/*.nc" --save /your_path_to_store_result/results.csv

Key Options

Option	Description
`PATTERN`	GLOB-like path (use `*` and `?` wildcards)
`--project`	Project name (e.g. `ik1017`). Optional; auto-detected.
`--add COL`	Show additional metadata columns (e.g. `uid` or `gid`)
`--filter`	SQL predicate, e.g. `"size > 1e6 AND atime_ms = '2025-11-01'"`
`--max_rows N`	Show at most N rows
`--save PATH`	Save results to `.parquet`, `.csv` or `.json`

limscaup-du#

limscaup-du provides fast, directory-level summaries similar to the UNIX du command. You can view total size, number of files, or aggregated metadata for any directory prefix.

Examples

Warning

You can only search within your own projects and files.

# Show total size of a directory in /work
limscaup-du /work/ik1017/ # adjust to one of your projects

# Limit display to largest subdirectories
limscaup-du /work/ik1017/ --max_rows 20

# Get additional information through added columns (e.g. uid)
limscaup-du /work/ik1017/ --add uid

Key Options

Option	Description
`PATH`	Path on a supported mount
`--filter`	SQL predicate applied before counting, e.g. `"atime_ms > '2025-06-15' AND size >= 5000"` (needs both quotes)
`--add col[:agg]`	Additional columns to aggregate (repeatable). Allowed aggs: `any`, `max`, `min`, `avg`, `sum`, `count`. Default: `any`.
`--max_rows N`	Limit number of rows shown for subdirectories
`--save PATH`	Save results; directory (Parquet) or file path with `.parquet` / `.csv` / `.json` to select format

Note

Size might vary slightly compared to du on /work due to block size instead of file size beeing used. As blocks are 4096 bytes, size of limscaup-du and limscaup-find will always be a multiple of this and might therefore be slightly bigger than the size plain du would report.

While du traverses the directory tree from a root inode, limscaup-du aggregates using matching paths strings. This allows to start from multiple roots matching a given substring. In order to start from a single directory root, append / to the provided path string to resemble the exact du. For example du /work/ik1017/CMIP6 would just traverse that directory while limscaup-du /work/ik1017/CMIP6 would also include /work/ik1017/CMIP6input/ and /work/ik1017/CMIP6_tmp/.