S3 Object Storage#

file version: 12 Dec 2024

Introduction#

The DKRZ provides an S3-compatible object storage system based on a MinIO deployment. This storage system closely resembles the widely used S3 storage by Amazon Web Services (AWS S3) in terms of structure and features. It allows to store large amounts of unstructured data and facilitates access through web-browsers, command line interfaces (CLIs), APIs, and data clients.

To efficiently manage data, the S3 storage employs objects, buckets and metadata.

Objects store data content, such as documents, images, and more.

Buckets act as containers for organizing and storing objects. Unlike traditional file systems, each bucket has a unique name across the entire storage system, simplifying data retrieval regardless of volume or location.

Metadata provides essential information about objects, enhancing data management through efficient search, retrieval, and organization.

Use Cases#

Object storage is ideal for storing static data—files that are written once and accessed multiple times without further modification. It’s well-suited for:

  • Distribution: Accessing data from various devices, including notebooks, smartphones, and supercomputers.

  • Sharing: Sharing large datasets with colleagues and for public access.

  • Backup: Store snapshots or copies of your datasets.

However, S3 storage is not suitable for storing dynamic data that undergoes frequent modifications or ongoing development.

Accounts and Quotas#

The quotas are set per bucket. This means that each user’s data usage will count towards the total bucket quota. At the current stage, each bucket has a fixed quota of:

  • 100 TiB per bucket

Requests for individual buckets to share larger data sets can be made to support@dkrz.de.

Interfaces#

S3 object storage can be accessed via the web-based interface or API at the following URLs:

Access to the S3 Storage via both interfaces is described in the subsequent chapters.

Further reading#

Questions#

For technical questions regarding DKRZ’s S3 object storage system, please contact support@dkrz.de

On this page