Python-swiftclient#

Python-swiftclient comprises a command line tool swift and Python APIs for accessing Swift object store. It supports the following platforms:

Linux/Unix
OS X
Windows

Swift Tool#

Usage#

Python-swiftclient is already installed on Levante. Before you can use the swift command line tool please follow these steps:

Load the appropriate environment module:
```
module load py-python-swiftclient
```
After loading the module you have to generate an authentication token by executing
```
swift-token new
```
This sets some environment variables used by swift as well as creating two files in your home directory (.swiftenv and .swiftenv.tcl). The next time you load the module your token will be re-used if it is still valid (tokens are valid for 30 days by default).
After creating a new token you have to unload and load the environment module again using the switch or swap module command:
```
module switch py-python-swiftclient/3.12.0-gcc-11.2.0
```

Reminder login token expires#

If you wish to be reminded when the login token is due to expire, you can set up SLURM job to check the expiration data of the login token on daily basis. You might use this script for this purpose. It sends an email to you when the token expires in 30, 5, 4, 3, 2 and 1 days. The email is send to email address which is stored in your user profile at https://luv.dkrz.de . The whole output is written into the file swift_token_expire_check.log which is automatically kept below a size of 1100 lines. When finished, the script submits itself to start again the next day at the same time. Please submit the script this way replacing YOUR_PROJECT_ACCOUNT by a valid project/account with compute time:

sbatch --begin="now+1day" --account=YOUR_PROJECT_ACCOUNT ./swift_token_expire_email.sh

Help#

To display the most important swift command line options use:

man swift
swift help

Please visit the official documentation for full reference.

Listing containers and objects#

To list all containers use:

swift list

To list all objects in container <containername> use:

swift list <containername>

Creating containers#

To create a new container <containername> (if it doesn’t exist) enter:

swift post <containername>

Uploading data#

To upload named files to container <containername> use:

swift upload <containername> file [file2, file3, ...]

Single objects are limited to 5 GB in size. If you want to upload larger files, you have to modify the above command:

swift upload --segment-size 1000000000 <containername> file [file2, file3, ...]

In this example the files are split into chunks of 1 GB size each and put into a container named <containername>_segments. After uploading all chunks a so-called manifest file with the original filename is created. If this object is downloaded, the server will stream all chunks together transparent for the user. Use the --segment-container option to specifiy the container where the chunks are placed:

swift upload --segment-size 1000000000 --segment-container <containername> <containername> file [file2, file3, ...]

This is especially useful when uploading data into the scratch space (dkrz_scratch). In order to put all the data into the scratch, you must set the segment-container to dkrz_scratch. This prevents that the segments are put in a separate container in your account, which would not allow you to use the space of the scratch. So, in order to use the scratch, use following command:

swift upload --segment-size 1000000000 --segment-container dkrz_scratch dkrz_scratch file [file2, file3, ...]

Downloading data#

To download all objects of all containers use:

swift download --all

To download all objects from container <containername> use:

swift download <containername>

To download the file object <objectname> from container <containername> enter:

swift download <containername> <objectname>

When uploading a file with the swift commandline tool, it preserves the current modified time (at least on Unix-like systems) with a metadata entry. When redownloading such a file, the modified time is used for creating the file on the local filesystem. This can lead to problems in certain directories (e.g. /tmp, /scratch) where files are automatically deleted if they are older than a certain time. Therefore, there exist an option to ignore the modified time stored on the server, so that fresh timestamps are used:

swift download --ignore-mtime <containername> <objectname>

Deleting data#

To delete all objects of all containers (be careful what you are doing, because all files are lost!) use:

swift delete --all

To delete all objects from container <containername> enter:

swift delete <containername>

To delete object <objectname> from container <containername> use:

swift delete <containername> <objectname>

Create temporary URL#

First, you need to retrieve the swift storage account name (not the LDAP name) and the secret key of your account. Use the stat command for this purpose

swift stat

If the secret temporary URL key does not already exist, create one with:

swift post -m "Temp-URL-Key:<key>"

If you want to create a download url, which is valid for one day, use the following command:

swift tempurl GET 86400 /v1/<account>/<containername>/<objectname> <key>

Prepend the server host https://swift.dkrz.de to the output, and send the url to the user. The user can then use for example curl for the download:

curl "https://swift.dkrz.de/v1/<account>/<containername>/<objectname>?\
temp_url_sig=8530f04a7611f6c860243b553d53a6aa1955c1f95&temp_url_expires=1438776977" > file

You can also use ISO 8601 timestamps instead of UNIX timestamps, e.g.:

swift tempurl GET 2017-07-25 /v1/<account>/<containername>/<objectname> <key>

If you want to allow a user to upload data, use PUT instead of GET

swift tempurl PUT 86400 /v1/<account>/<containername>/<objectname> <key>

If the user uses curl for upload, he must specify the method with the -X switch:

curl "https://swift.dkrz.de/v1/<account>/<containername>/<objectname>?\
temp_url_sig=8530f04a7611f6c860243b553d53a6aa1955c1f95&temp_url_expires=1438776977" \
-XPUT --data-binary @file

You can also create the temporary URL using Python (https://docs.openstack.org/swift/xena/api/temporary_url_middleware.html).

Prefix-based temporary URL#

You can create one temporary URL which is valid for all object names which start with a common prefix. This is useful for sharing a whole container or pseudofolder (= objects with same prefix in the name):

swift tempurl --prefix-based PUT 86400 /v1/<account>/<containername>/<prefix> <key>

A user who wants to upload data into the pseudofolder must inject the object name (which must start with the same prefix) into the generated URL:

https://swift.dkrz.de/v1/<account>/<containername>/<objectname>?temp_url_sig=8530f04a7611f6c860243b553d53a6aa1955c1f95
&temp_url_expires=1438776977&temp_url_prefix=<prefix>

If you want to share a whole container, you must use an empty prefix.

Python Interface#

Alongside the swift command line tool, python-swiftclient includes APIs to access the Swift object store from a Python program. As a prerequisite, you need to load the python-swiftclient module by typing:

module load py-python-swiftclient

The following example shows how to create a container and upload a file using swiftclient Python module:

from swiftclient.client import Connection
from getpass import getpass

pw = getpass()

c = Connection("https://swift.dkrz.de/auth/v1.0", "account:user", pw, auth_version=1.0)

c.put_container("my_container")

f = open("/path/to/myfile")

c.put_object("my_container", "my_object", f)

To retrieve all objects and metadata from a container:

# Credentials, and imports
...
metadata, objects = c.get_container("my_container")

Installing python-swiftclient on your local system#

The recommended method is to use pip:

pip install python-swiftclient

You need to generate a token for authentication. We have developed a python script for this purpose. You can download it here. If you do not want to use the script, you need to specify your credentials for each request:

swift -U <account>:<username> -A https://swift.dkrz.de/auth/v1.0 -K <key> stat