Python-swiftclient

Python-swiftclient comprises a command line tool swift and Python APIs for accessing Swift object store. It supports the following platforms:

  • Linux/Unix

  • OS X

  • Windows

Swift Tool

Usage

Python-swiftclient is already installed on Mistral and Levante. Before you can use the swift command line tool please follow these steps:

  1. Load the appropriate environment module. Please, note that module names differ on Mistral and Levante:

    # mistral
    module load swift
    
    # levante
    module load py-python-swiftclient
    
  2. After loading the module you have to generate an authentication token by executing

    swift-token new
    

    This sets some environment variables used by swift as well as creating two files in your home directory (.swiftenv and .swiftenv.tcl). The next time you load the module your token will be re-used if it is still valid (tokens are valid for 7 days by default).

  3. After creating a new token you have to load the environment module again:

    # mistral
    module load swift
    
    # levante
    module load py-python-swiftclient
    

Note

In case you run into problems using the swift client tool on Levante, unloading and loading the environment module again can often fix these problems:

module unload py-python-swiftclient
module load py-python-swiftclient

Help

To display the most important swift command line options use:

man swift
swift help

Please visit the official documentation for full reference.

Listing containers and objects

To list all containers use:

swift list

To list all objects in container <containername> use:

swift list <containername>

Creating containers

To create a new container <containername> (if it doesn’t exist) enter:

swift post <containername>

Uploading data

To upload named files to container <containername> use:

swift upload <containername> file [file2, file3, ...]

Single objects are limited to 5 GB in size. If you want to upload larger files, you have to modify the above command:

swift upload --segment-size 1000000000 <containername> file [file2, file3, ...]

In this example the files are split into chunks of 1 GB size each and put into a container named <containername>_segments. After uploading all chunks a so-called manifest file with the original filename is created. If this object is downloaded, the server will stream all chunks together transparent for the user. Use the --segment-container option to specifiy the container where the chunks are placed:

swift upload --segment-size 1000000000 --segment-container <containername> <containername> file [file2, file3, ...]

This is especially useful when uploading data into the scratch space (dkrz_scratch). In order to put all the data into the scratch, you must set the segment-container to dkrz_scratch. This prevents that the segments are put in a separate container in your account, which would not allow you to use the space of the scratch. So, in order to use the scratch, use following command:

swift upload --segment-size 1000000000 --segment-container dkrz_scratch dkrz_scratch file [file2, file3, ...]

Downloading data

To download all objects of all containers use:

swift download --all

To download all objects from container <containername> use:

swift download <containername>

To download the file object <objectname> from container <containername> enter:

swift download <containername> <objectname>

When uploading a file with the swift commandline tool, it preserves the current modified time (at least on Unix-like systems) with a metadata entry. When redownloading such a file, the modified time is used for creating the file on the local filesystem. This can lead to problems in certain directories (e.g. /tmp, /scratch) where files are automatically deleted if they are older than a certain time. Therefore, there exist an option to ignore the modified time stored on the server, so that fresh timestamps are used:

swift download --ignore-mtime <containername> <objectname>

Deleting data

To delete all objects of all containers (be careful what you are doing, because all files are lost!) use:

swift delete --all

To delete all objects from container <containername> enter:

swift delete <containername>

To delete object <objectname> from container <containername> use:

swift delete <containername> <objectname>

Share container via ACL

You can share a container via an Access Control List (ACL). For this purpose, you have to add metadata to the container, containing the access right and the account and user, to whom access should be granted.

If you want to grant read access to your container <containername> to the user <user> belonging to the account <account>, you add the read ACL via

swift post -r <account>:<user> <containername>

If you want to grant access to all users of account <account>, just skip the user part:

swift post -r <account> <containername>

The user can then access the container specifiying your storage url. You must inform the user with your own swift storage account name, which you retrieve via the command:

swift stat

The user then accesses the container via:

swift --os-storage-url "https://swift.dkrz.de/v1/<account>" list <container>

If you want to grant write access, use the -w instead of -r switch (or both).

Create temporary URL

First, you need to retrieve the swift storage account name (not the LDAP name) and the secret key of your account. Use the stat command for this purpose

swift stat

If the secret temporary URL key does not already exist, create one with:

swift post -m "Temp-URL-Key:<key>"

If you want to create a download url, which is valid for one day, use the following command:

swift tempurl GET 86400 /v1/<account>/<containername>/<objectname> <key>

Prepend the server host https://swift.dkrz.de to the output, and send the url to the user. The user can then use for example curl for the download:

curl "https://swift.dkrz.de/v1/<account>/<containername>/<objectname>?\
temp_url_sig=8530f04a7611f6c860243b553d53a6aa1955c1f95&temp_url_expires=1438776977" > file

You can also use ISO 8601 timestamps instead of UNIX timestamps, e.g.:

swift tempurl GET 2017-07-25 /v1/<account>/<containername>/<objectname> <key>

If you want to allow a user to upload data, use PUT instead of GET

swift tempurl PUT 86400 /v1/<account>/<containername>/<objectname> <key>

If the user uses curl for upload, he must specify the method with the -X switch:

curl "https://swift.dkrz.de/v1/<account>/<containername>/<objectname>?\
temp_url_sig=8530f04a7611f6c860243b553d53a6aa1955c1f95&temp_url_expires=1438776977" \
-XPUT --data-binary @file

You can also create the temporary URL using Python (https://docs.openstack.org/swift/xena/api/temporary_url_middleware.html).

Prefix-based temporary URL

You can create one temporary URL which is valid for all object names which start with a common prefix. This is useful for sharing a whole container or pseudofolder (= objects with same prefix in the name):

swift tempurl --prefix-based PUT 86400 /v1/<account>/<containername>/<prefix> <key>

A user who wants to upload data into the pseudofolder must inject the object name (which must start with the same prefix) into the generated URL:

https://swift.dkrz.de/v1/<account>/<containername>/<objectname>?temp_url_sig=8530f04a7611f6c860243b553d53a6aa1955c1f95
&temp_url_expires=1438776977&temp_url_prefix=<prefix>

If you want to share a whole container, you must use an empty prefix.

Python Interface

Alongside the swift command line tool, python-swiftclient includes APIs to access the Swift object store from a Python program. As a prerequisite, you need to load the python-swiftclient module by typing:

# mistral
module load swift

# levante
module load py-python-swiftclient

The following example shows how to create a container and upload a file using swiftclient Python module:

from swiftclient.client import Connection
from getpass import getpass

pw = getpass()

c = Connection("https://swift.dkrz.de/auth/v1.0", "account:user", pw, auth_version=1.0)

c.put_container("my_container")

f = open("/path/to/myfile")

c.put_object("my_container", "my_object", f)

To retrieve all objects and metadata from a container:

# Credentials, and imports
...
metadata, objects = c.get_container("my_container")

Installing python-swiftclient on your local system

The recommended method is to use pip:

pip install python-swiftclient

You need to generate a token for authentication. We have developed a python script for this purpose. You can download it here. If you do not want to use the script, you need to specify your credentials for each request:

swift -U <account>:<username> -A https://swift.dkrz.de/auth/v1.0 -K <key> stat