Python-swiftclient comprises a command line tool swift and Python APIs for accessing Swift object store. It supports the following platforms:

  • Linux/Unix

  • OS X

  • Windows

Swift Tool#


Python-swiftclient is already installed on Levante. Before you can use the swift command line tool please follow these steps:

  1. Load the appropriate environment module:

    module load py-python-swiftclient
  2. After loading the module you have to generate an authentication token by executing

    swift-token new

    This sets some environment variables used by swift as well as creating two files in your home directory (.swiftenv and .swiftenv.tcl). The next time you load the module your token will be re-used if it is still valid (tokens are valid for 30 days by default).

  3. After creating a new token you have to unload and load the environment module again using the switch or swap module command:

    module switch py-python-swiftclient/3.12.0-gcc-11.2.0

Reminder login token expires#

If you wish to be reminded when the login token is due to expire, you can set up SLURM job to check the expiration data of the login token on daily basis. You might use this script for this purpose. It sends an email to you when the token expires in 30, 5, 4, 3, 2 and 1 days. The email is send to email address which is stored in your user profile at . The whole output is written into the file swift_token_expire_check.log which is automatically kept below a size of 1100 lines. When finished, the script submits itself to start again the next day at the same time. Please submit the script this way replacing YOUR_PROJECT_ACCOUNT by a valid project/account with compute time:

sbatch --begin="now+1day" --account=YOUR_PROJECT_ACCOUNT ./


To display the most important swift command line options use:

man swift
swift help

Please visit the official documentation for full reference.

Listing containers and objects#

To list all containers use:

swift list

To list all objects in container <containername> use:

swift list <containername>

Creating containers#

To create a new container <containername> (if it doesn’t exist) enter:

swift post <containername>

Uploading data#

To upload named files to container <containername> use:

swift upload <containername> file [file2, file3, ...]

Single objects are limited to 5 GB in size. If you want to upload larger files, you have to modify the above command:

swift upload --segment-size 1000000000 <containername> file [file2, file3, ...]

In this example the files are split into chunks of 1 GB size each and put into a container named <containername>_segments. After uploading all chunks a so-called manifest file with the original filename is created. If this object is downloaded, the server will stream all chunks together transparent for the user. Use the --segment-container option to specifiy the container where the chunks are placed:

swift upload --segment-size 1000000000 --segment-container <containername> <containername> file [file2, file3, ...]

This is especially useful when uploading data into the scratch space (dkrz_scratch). In order to put all the data into the scratch, you must set the segment-container to dkrz_scratch. This prevents that the segments are put in a separate container in your account, which would not allow you to use the space of the scratch. So, in order to use the scratch, use following command:

swift upload --segment-size 1000000000 --segment-container dkrz_scratch dkrz_scratch file [file2, file3, ...]

Downloading data#

To download all objects of all containers use:

swift download --all

To download all objects from container <containername> use:

swift download <containername>

To download the file object <objectname> from container <containername> enter:

swift download <containername> <objectname>

When uploading a file with the swift commandline tool, it preserves the current modified time (at least on Unix-like systems) with a metadata entry. When redownloading such a file, the modified time is used for creating the file on the local filesystem. This can lead to problems in certain directories (e.g. /tmp, /scratch) where files are automatically deleted if they are older than a certain time. Therefore, there exist an option to ignore the modified time stored on the server, so that fresh timestamps are used:

swift download --ignore-mtime <containername> <objectname>

Deleting data#

To delete all objects of all containers (be careful what you are doing, because all files are lost!) use:

swift delete --all

To delete all objects from container <containername> enter:

swift delete <containername>

To delete object <objectname> from container <containername> use:

swift delete <containername> <objectname>

Share container via ACL#

You can share a container via an Access Control List (ACL). For this purpose, you have to add metadata to the container, containing the access right and the account and user, to whom access should be granted.

If you want to grant read access to your container <containername> to the user <user> belonging to the account <account>, you add the read ACL via

swift post -r <account>:<user> <containername>

If you want to grant access to all users of account <account>, just skip the user part:

swift post -r <account> <containername>

The user can then access the container specifiying your storage url. You must inform the user with your own swift storage account name, which you retrieve via the command:

swift stat

The user then accesses the container via:

swift --os-storage-url "<account>" list <container>

If you want to grant write access, use the -w instead of -r switch (or both).

Create temporary URL#

First, you need to retrieve the swift storage account name (not the LDAP name) and the secret key of your account. Use the stat command for this purpose

swift stat

If the secret temporary URL key does not already exist, create one with:

swift post -m "Temp-URL-Key:<key>"

If you want to create a download url, which is valid for one day, use the following command:

swift tempurl GET 86400 /v1/<account>/<containername>/<objectname> <key>

Prepend the server host to the output, and send the url to the user. The user can then use for example curl for the download:

curl "<account>/<containername>/<objectname>?\
temp_url_sig=8530f04a7611f6c860243b553d53a6aa1955c1f95&temp_url_expires=1438776977" > file

You can also use ISO 8601 timestamps instead of UNIX timestamps, e.g.:

swift tempurl GET 2017-07-25 /v1/<account>/<containername>/<objectname> <key>

If you want to allow a user to upload data, use PUT instead of GET

swift tempurl PUT 86400 /v1/<account>/<containername>/<objectname> <key>

If the user uses curl for upload, he must specify the method with the -X switch:

curl "<account>/<containername>/<objectname>?\
temp_url_sig=8530f04a7611f6c860243b553d53a6aa1955c1f95&temp_url_expires=1438776977" \
-XPUT --data-binary @file

You can also create the temporary URL using Python (

Prefix-based temporary URL#

You can create one temporary URL which is valid for all object names which start with a common prefix. This is useful for sharing a whole container or pseudofolder (= objects with same prefix in the name):

swift tempurl --prefix-based PUT 86400 /v1/<account>/<containername>/<prefix> <key>

A user who wants to upload data into the pseudofolder must inject the object name (which must start with the same prefix) into the generated URL:<account>/<containername>/<objectname>?temp_url_sig=8530f04a7611f6c860243b553d53a6aa1955c1f95

If you want to share a whole container, you must use an empty prefix.

Python Interface#

Alongside the swift command line tool, python-swiftclient includes APIs to access the Swift object store from a Python program. As a prerequisite, you need to load the python-swiftclient module by typing:

module load py-python-swiftclient

The following example shows how to create a container and upload a file using swiftclient Python module:

from swiftclient.client import Connection
from getpass import getpass

pw = getpass()

c = Connection("", "account:user", pw, auth_version=1.0)


f = open("/path/to/myfile")

c.put_object("my_container", "my_object", f)

To retrieve all objects and metadata from a container:

# Credentials, and imports
metadata, objects = c.get_container("my_container")

Installing python-swiftclient on your local system#

The recommended method is to use pip:

pip install python-swiftclient

You need to generate a token for authentication. We have developed a python script for this purpose. You can download it here. If you do not want to use the script, you need to specify your credentials for each request:

swift -U <account>:<username> -A -K <key> stat