Python

We provide several precompiled Python environments on Levante but you can also create your own environments.

System wide Python installations

The environment modules for our system wide installations follow the naming convention python3/YYYY.XX-compilerversion. One such module would be python3/2023.01-gcc-11.2.0. It can can be accessed as follows:

module load python3/2023.01-gcc-11.2.0

Python environments are updated about every half year. The most recent recommended version can be loaded by just stating:

module load python3

However, we recommend to always use the full module name to avoid surprises when we switch to a new default environment.

New and updated packages will only be installed into the most recent installation. These updates can break your existing scripts based on python3. Use an older (stable) version to avoid this possibility.

Individual Python installation

We often see users who try to add custom python packages on-top of our globally provided python environments, if certain packages or package versions are missing. If done outside of a local (conda or virtual) environment using pip, these installations will, by default, end up in the user’s home folder ~/.local/lib/python*. These default paths are searched by python, independently of the python module loaded, and preferred over the packages installed in the environment. In some cases this might work, but in many cases when an incompatible package version is loaded, this will result in hard-to-debug error messages, that are not reproducible for other users.

Usually, we recommend to remove these pip installations by simply deleting ~/.local/lib/python*. However, it should be noted that deleting these folders might also delete installed python packages that were not conflicting, and hence potentially break other workflows. Do this only on your own risk!

To prevent this problem from occurring and to have full control over every aspect of your Python environment, please use conda (or virtual) environments to encapsulate your local changes. In the following a brief introduction, how to setup and use conda is given.

Set up conda for individual environments

Conda ships with many useful defaults but some configuration should be adapted for best results on Levante. This has to be done only once.

You can start with the conda command from our python3 modules.

module load python3

Location of conda environments and packages

The default location for your own environments and packages is in your home directory (~/.conda). Because of the small quota you have there, it makes sense to move the location for these files to /work.

Set the new location before creating any environments.

conda config --add envs_dirs /work/bb5555/u555555/conda/envs
conda config --add pkgs_dirs /work/bb5555/u555555/conda/pkgs

Please fill in the work location you want to use for your environments.

Initialization

Before you add the initialization, you should prevent conda from activating its base environment by default.

conda config --set auto_activate_base false

Then you can go ahead and init conda.

conda init

This will add initialization code to your ~/.bashrc file if your shell is bash. Note that .bashrc is normally not read when the shell is a login shell.

If you want the conda initialization to be performed also for login shells, then you should move the code to ~/.profile for example.

You can also leave it in .bashrc and only source it on demand in login shells

. ~/.bashrc

An alternative is to forgo conda init and just run

eval "$(conda shell.bash hook)"

whenever you need to activate your conda enviroments.

Basic conda commands

Create an environment named my_env and install the jupyter package into it:

conda create --name my_env jupyter

Show available environments:

conda info --env

Activate an environment:

conda activate my_env

If you do not initialize conda as described above, conda activate ... will produce an error. In this case, you can use source activate ... or run eval "$(conda shell.bash hook)" before using conda activate ...

Show installed conda packages:

conda list

Install additional package(s):

conda install numpy

You can also use the mamba command which is much faster than conda in many cases.

mamba install numpy

More information on conda can be found in the user guide.

JupyterHub

JupyterHub is a multi-user server for Jupyter notebooks that allows to execute the notebook directly on the DKRZ HPC system Levante. It also supports the execution of parallel computation.

JupyterHub is available at https://jupyterhub.dkrz.de for all DKRZ users who have access to Levante and who are allowed to submit batch jobs.

You can use your mobile device or the workstation in your office. Windows or MacOS or Linux all just work. For more information see our JupyterHub documentation.