Swift Object Storage#
The distributed object storage system OpenStack Swift operated by DKRZ can be used to store and share data in various ways:
Access data from different devices, e.g. notebook, smartphone and supercomputer
Share data with colleagues with or without an account at DKRZ
Replicate data between different sites automatically
Please notice that all files are kept in three copies on different servers, but there is NO backup from which you could restore deleted or damaged files!
Access to Swift is provided for each user which is active in at least one project. There are two types of accounts available: user accounts and project accounts. User accounts are accessible only by the user himself whereas project accounts are shared between all project members.
50 GiB per user
10 TiB per project
We provide a scratch space of about 600 TiB in total. Please be
aware that data put into this space will be automatically deleted
after 2 weeks! You find the scratch in a container called
dkrz_scratch in your account, but only if your account is used
(which means at least one container has already been created).
Interfaces to Swift#
swift - a command line tool for Linux, Unix, OS X and Windows, provided as part of python-swiftclient
swiftclient - Python API which forms another part of python-swiftclient
S3 API (if access with the S3 protocol is desired, please contact firstname.lastname@example.org)
If you use the RESTful API, you first need to obtain a token. For
this, you have to make a
GET request to
https://swift.dkrz.de/auth/v1.0 with the headers
How it works#
Please be aware that Swift is a so called “object system”, working similar to Amazon S3. The Object Storage system organizes data in a hierarchy, as follows:
- Account: Represents the top-level of the hierarchy. The account
defines a namespace for containers. A container might have the same name in two different accounts.
- Container: Defines a namespace for objects. An object with the
same name in two different containers represents two different objects. You can create any number of containers within an account. In addition to containing objects, you can also use the container to control access to objects by using an access control list (ACL). You cannot store an ACL with individual objects.
- Object: Stores data content, such as documents, images, and so
on. You can also store custom metadata with an object.
It is important to understand that an object storage system is not
used like a POSIX filesystem. For example, there is no concept of a
directory. If you want to create a hierachy inside a container, you
must include the hierarchy name in your object names. For example, if
you want to distinguish between 2 hierarchies
could call the objects for the first hierarchy
/a/object2, etc and for the second
etc. Furthermore, moving (without copying) an object from one
container to another or renaming an object is not
possible. Moving/renaming always includes a (potential) expensive copy
For technical questions regarding DKRZ’s object storage system, please contact email@example.com.