Storage Access#

file version: 12 Dec 2024

Web-based Interface#

Access to the S3 Object Storage is available via web browser by logging into the web-based interface.

Once logged in, navigation through buckets, personal sub-folders, and shared sub-folders can be done using the Object Browser. Data can be stored, previewed, deleted, and downloaded using the web-based interface.

While access to the object storage is provided through the web-based interface, integration into scripts and computing environments is enabled by leveraging the API.

To utilize the API with tools like the MinIO client, access key pairs can be generated as outlined in the following section.

Generating Access Keys#

In the web-based interface, navigate to the ‘Access Keys’ section.

The access key pair initially grants both read and write access to the personal sub-folder within the bucket.

Further customization of access rights can be done by activating the ‘Restrict beyond user policy’ feature, allowing custom permissions to be set.

These settings adhere to the JSON-formatted IAM policy explained here.

Note: Access key pair data (<ACCESS_KEY> + <SECRET_KEY>) should be documented or downloaded in the next step, as the <SECRET_KEY> cannot be displayed again for security reasons.

Additionally, it’s good practice to assign a name to each key pair for identification in the future, particularly if separate keys are used across different machines.

Access via Command Line#

To access the S3 storage via the command line, the MinIO client (CLI command mc) can be used.

On Levante, the MinIO client is pre-installed as the minio-client module. The module can be loaded with the following command:

module load minio-client

Note: The minio-client module should not be confused with the Midnight Commander module (module name mc).

For installation of mc on a personal machine, the download and quickstart guide can be found here.

To access the storage system, create an <ALIAS> using the previously generated key pair (<ACCESS_KEY> and <SECRET_KEY>):

mc alias set <ALIAS> https://s3.eu-dkrz-1.dkrz.cloud <ACCESS_KEY> <SECRET_KEY>

Replace:

  • <ALIAS> with a suitable name, such as S3DKRZ.

  • <ACCESS_KEY> with the access key from the key pair.

  • <SECRET_KEY> with the secret key from the key pair.

Once the alias is set, it can be used to access a <BUCKET>. For instance, to list the content of <BUCKET>:

mc ls <ALIAS>/<BUCKET>

To list the content of a sub-folder belonging to a <USER_ID>:

mc ls <ALIAS>/<BUCKET>/<USER_ID>

To copy a <FILE> into the respective subfolder:

mc cp <FILE> <ALIAS>/<BUCKET>/<USER_ID>

For more details and commands, refer to the MinIO-mc documentation.

Note: Mounting the S3 storage via software like s3fs should be avoided, as it results in an excessively high number of active connections.

GUI Client#

For a graphical user interface (GUI) client option, CrossFTP can be utilized.

On this page