S3 Object Storage#

S3 Object Storage is currently in the in-house testing phase. It can therefore only be accessed from the DKRZ’s internal network. Please use with caution.

Use Cases#

The DKRZ provides an S3-compatible object storage system based on a MinIO deployment. The S3 storage system allows for:

  • Accessing data from different devices, such as notebooks, smartphones, and supercomputers.

  • Sharing data with colleagues, with or without an account at DKRZ.

  • Replicating data between different sites.

Accounts#

Access to the S3 storage is provided for each user who is a member of a DKRZ department. For each department, an individual bucket is deployed. Within these buckets, each active user will find a protected subdirectory in which they can store their data, accessible only to them. Users can also grant access to other members and third parties through individual rights management. Additionally, each user can generate user-specific access key pairs for authentication.

Quota#

There is a general bucket quota per department. The usage of all users of a department counts towards this quota, so that in an extreme case, a single user can use up the entire quota of the department bucket.

  • 100 TiB per department

Interfaces#

How it Works#

To access and transfer data to the S3 storage using the API, you’ll need to create an access key pair.

Log into the user-interface using your DKRZ credentials and navigate to the ‘Access Keys’ section. The access key pair initially provides both read and write access to your folder within the department bucket. You can customize access rights further by activating the ‘Restrict beyond user policy’ feature, which allows you to set custom permissions. This process follows a JSON-formatted IAM policy explained here.

Note: Remember to document or download the access key pair data (<ACCESS_KEY> + <SECRET_KEY>) in the next step, as the <SECRET_KEY> cannot be displayed again for security reasons.

Further, it’s a good practice to assign each key a name for easy identification in the future, especially if you use separate keys across different machines.

Access via Command Line#

To access S3 storage via the command line, you’ll need the MinIO client mc. You can find the download and quickstart guide here.

To access, create an <ALIAS> for the DKRZ S3 storage using the previously generated key pair (<ACCESS_KEY> and <SECRET_KEY>):

mc alias set <ALIAS> https://s3.eu-dkrz-1.dkrz.cloud <ACCESS_KEY> <SECRET_KEY>

Replace:

  • <ALIAS> with a suitable name, such as S3DKRZ.

  • <ACCESS_KEY> with the access key from the key pair.

  • <SECRET_KEY> with the secret key from the key pair.

Once the alias is set, you can use it to access a department <BUCKET>. For instance, to list the content of a specific <BUCKET>:

mc ls <ALIAS>/<BUCKET>

Currently, the following department buckets exist:

  • anwendung

  • datenanalyse

  • datenmanagement

  • system

To list the content of a subfolder belonging to a <USER_ID> (kXXXXXX):

mc ls <ALIAS>/<BUCKET>/<USER_ID>

To copy a <FILE> into the respective subfolder:

mc cp <FILE> <ALIAS>/<BUCKET>/<USER_ID>

For more details and commands, refer to the minio-mc documentation.

Note: Avoid mounting the S3 storage via software like s3fs, as it results in an excessively high number of active connections.

GUI Client#

For a graphical user interface (GUI) client option, CrossFTP can be utilized.

Questions#

For technical questions regarding DKRZ’s S3 object storage system, please contact support@dkrz.de.

On this page