Swift Object Storage#

Use Cases#

The distributed object storage system OpenStack Swift operated by DKRZ can be used to store and share data in various ways:

  • Access data from different devices, e.g. notebook, smartphone and supercomputer

  • Share data with colleagues with or without an account at DKRZ

  • Replicate data between different sites automatically

Please notice that all files are kept in three copies on different servers, but there is NO backup from which you could restore deleted or damaged files!

Accounts#

Access to Swift is provided for each user which is active in at least one project. There are two types of accounts available: user accounts and project accounts. User accounts are accessible only by the user himself whereas project accounts are shared between all project members.

Quota#

  • 50 GiB per user

  • 10 TiB per project

Scratch#

We provide a scratch space of about 600 TiB in total. Please be aware that data put into this space will be automatically deleted after 2 weeks! You find the scratch in a container called dkrz_scratch in your account, but only if your account is used (which means at least one container has already been created).

Interfaces to Swift#

If you use the RESTful API, you first need to obtain a token. For this, you have to make a GET request to https://swift.dkrz.de/auth/v1.0 with the headers X-Auth-User: <account>:<user> and X-Auth-Key: <password>.

How it works#

Please be aware that Swift is a so called “object system”, working similar to Amazon S3. The Object Storage system organizes data in a hierarchy, as follows:

  • Account: Represents the top-level of the hierarchy. The account

    defines a namespace for containers. A container might have the same name in two different accounts.

  • Container: Defines a namespace for objects. An object with the

    same name in two different containers represents two different objects. You can create any number of containers within an account. In addition to containing objects, you can also use the container to control access to objects by using an access control list (ACL). You cannot store an ACL with individual objects.

  • Object: Stores data content, such as documents, images, and so

    on. You can also store custom metadata with an object.

It is important to understand that an object storage system is not used like a POSIX filesystem. For example, there is no concept of a directory. If you want to create a hierachy inside a container, you must include the hierarchy name in your object names. For example, if you want to distinguish between 2 hierarchies a and b, you could call the objects for the first hierarchy /a/object1, /a/object2, etc and for the second /b/object1, /b/object2, etc. Furthermore, moving (without copying) an object from one container to another or renaming an object is not possible. Moving/renaming always includes a (potential) expensive copy operation.

Questions#

For technical questions regarding DKRZ’s object storage system, please contact support@dkrz.de.

On this page