Python#
We provide several precompiled Python environments on Levante but you can also create your own environments.
System wide Python installations#
The environment modules for our system wide installations follow the naming
convention python3/YYYY.XX-compilerversion. One such module would be
python3/2023.01-gcc-11.2.0. It can can be accessed as follows:
module load python3/2023.01-gcc-11.2.0
Python environments are updated about every half year. The most recent recommended version can be loaded by just stating:
module load python3
However, we recommend to always use the full module name to avoid surprises when we switch to a new default environment.
New and updated packages will only be installed into the most recent
installation. These updates can break your existing scripts based on
python3. Use an older (stable) version to avoid this possibility.
The python environments loaded by the python3 module files are CPU-only
and do not contain GPU-enabled packages. A globally installed python
environment that has GPU support for some packages (e.g. pytorch and
tensorflow) is available by loading the module pytorch.
Individual Python installation#
We often see users who try to add custom python packages on-top of our globally provided python environments, if certain packages or package versions are missing. If done outside of a local (conda or virtual) environment using pip, these installations will, by default, end up in the user’s home folder ~/.local/lib/python*. These default paths are searched by python, independently of the python module loaded, and preferred over the packages installed in the environment. In some cases this might work, but in many cases when an incompatible package version is loaded, this will result in hard-to-debug error messages, that are not reproducible for other users.
Usually, we recommend to remove these pip installations by simply deleting ~/.local/lib/python*. However, it should be noted that deleting these folders might also delete installed python packages that were not conflicting, and hence potentially break other workflows. Do this only on your own risk!
To prevent this problem from occurring and to have full control over every aspect of your Python environment, please use conda (or virtual) environments to encapsulate your local changes. In the following a brief introduction, how to setup and use conda is given.
Set up conda for individual environments#
Conda ships with many useful defaults but some configuration should be adapted for best results on Levante. This has to be done only once.
You can start with the conda command from our python3
modules.
module load python3
Warning
Conda channels are the locations where packages are
stored. You should only use the conda-forge channel which is
free for all to use. We would be very grateful if you could ensure
that only free and open channels are always used even for locally
installed conda.
All conda environments created from the system wide python modules
use the conda-forge channel. In rare cases, where you need to
add additional channels, you have to make sure that the license
allows free for all use.
Miniconda, by default, uses a channel which may require a paid
license. You are not allowed to use miniconda unless you remove the
default channel and replace it with conda-forge.
Location of conda environments and packages#
The default location for your own environments and packages is in your
home directory (~/.conda). Because of the small quota you have there, it makes
sense to move the location for these files to /work.
Set the new location before creating any environments.
conda config --add envs_dirs /work/bb5555/u555555/conda/envs
conda config --add pkgs_dirs /work/bb5555/u555555/conda/pkgs
Please fill in the work location you want to use for your environments.
Initialization#
Before you add the initialization, you should prevent conda from activating its base environment by default.
conda config --set auto_activate_base false
Then you can go ahead and init conda.
conda init
This will add initialization code to your ~/.bashrc file if your shell
is bash. Note that .bashrc is normally not read when the shell is a
login shell.
If you want the conda initialization to be performed also
for login shells, then you should move the code to ~/.profile for
example.
You can also leave it in .bashrc and only source it on demand in login shells
. ~/.bashrc
An alternative is to forgo conda init and just run
eval "$(conda shell.bash hook)"
whenever you need to activate your conda enviroments.
Basic conda commands#
Create an environment named my_env and install the jupyter package into it:
conda create --name my_env jupyter
Show available environments:
conda info --env
Activate an environment:
conda activate my_env
If you do not initialize conda as described above, conda activate ...
will produce an error. In this case, you can use source activate ...
or run eval "$(conda shell.bash hook)" before using
conda activate ...
Show installed conda packages:
conda list
Install additional package(s):
conda install numpy
You can also use the mamba command which is much faster than conda in many cases.
mamba install numpy
More information on conda can be found in the user guide.
Python using GPUs#
There exist several python packages that can be build with GPU support. As already mentioned above, some are already available via the public pytorch module.
For some packages / package versions it might be necessary to install it locally using for example conda.
How to do this is, however, not always straight-forward and depends on the package itself, its version and dependencies and cuda version and generally requires some online research.
If you struggle installing your GPU-enabled python environment get in touch with us.
JupyterHub#
JupyterHub is a multi-user server for Jupyter notebooks that allows to execute the notebook directly on the DKRZ HPC system Levante. It also supports the execution of parallel computation.
JupyterHub is available at https://jupyterhub.dkrz.de for all DKRZ users who have access to Levante and who are allowed to submit batch jobs.
You can use your mobile device or the workstation in your office. Windows or MacOS or Linux all just work. For more information see our JupyterHub documentation.