Kernels

Kernels are programming language specific processes that run independently and interact with the Jupyter Applications and their user interfaces 1.

Here you are going to learn how to use default kernels and how to enable your customized environments based on e.g conda or virtualenv. Please contact support@dkrz.de for any issue or request.

System-wide kernels

Note

Most of the kernels/modules are based on conda environments, after activation you can use conda list to check all installed packages and their versions.

Levante

Currently, we provide a Python 3 kernel based on the python3/2022.01-gcc-11.2.0 module. Widely used open-source packages are already installed. We will/might add more when corresponding modules are available.

Note

Currenty, there are many open requests for new softwares on Levante. It might take a while to install new packages and extensions, we encourage you to use your own conda environment.

Mistral

On Mistral, we povide the following Python kernels which are based on Python modules:

  • Python 3 / Unstable
    • it always point to the latest python module

    • based on this new concept (see details)

    • contains most used packages such as: xarray, dask, tensorflow, matplotlib, numpy

  • R 4.0.3

  • Julia 1.5.2 (using the module julia/1.5.2-gcc-9.1.0)

  • ESMValTool (based on the latest esmvaltool module, currently 2.2.0)

From November 15, 2021, the following kernels become deprecated (see Link1 and Link2):

  • Python 3 bleeding edge (using the module anaconda3/bleeding_edge)

  • Python 3 (using the module Python/3.5.2)

  • Python 2 bleeding_edge (using the module anaconda2/bleeding_edge)

  • Python 2 (using the module python/2.7.12)

Most of these kernels/modules are based on conda, so you can use conda list to check all installed packages and their versions.

Note

You can not install or update packages in the system-wide modules. We do not recommend using the --user flag to install new packages in your $HOME directory, it will not always work.

Note

For testing new libraries/packages, we suggest that you use your own conda/env as described below and you turn it into a kernel.

Wrapper packages

Some Python libraries/kernels are just wrappers arround software binaries e.g CDO. In this case, the binary needs to be loaded before using the wrapper, otherwise you will get the Module ‘xyz’ not found error. In Python 3 kernel some well known binaries are already loaded: CDO, SLK, git.

For any other wrapper, you can load modules or set environment variables in a file named .kernel_env (that you need to create in your home directory). This file is sourced anytime you start the default Python 3 kernel.

For instance, pynco requires the module netCDF Operator (NCO). You can load by adding this line to the .kernel_env file:

module load nco

Use your own kernel

To get full control of the Python interpreter and packages, we recommend that you create your own environment. Please, follow these steps:

Warning

Some users on Mistral are experiencing issues with Miniconda and Python 3.9, we recommend, that you first module load python3 and then create your conda/virtualenv environment.

  • With conda

% module load python3
% conda create -n env-name ipykernel python=3.x
% source activate env-name
% python -m ipykernel install --user --name my-kernel --display-name="My Kernel"
% conda deactivate
  • With virtualenv

If virtualenv is not available, you have to install it first before trying the following steps. The best way to install virtualenv is with pip:

% module load python3
% python -m pip install --user virtualenv
% python -m virtualenv --system-site-packages /path/to/new-kernel
% source /path/to/new-kernel/bin/activate
% pip install ipykernel
% python -m ipykernel install --user --name my-kernel --display-name="New Kernel"

You can now add/install additional packages that you need in your new environment and then:

(new-kernel) % deactivate

Finally:

  • (Re)start the server (jupyter notebook)

  • Refresh the browser (jupyterlab)

Now, the new kernel should be available.

Kernel specifications are in ~/.local/share/jupyter/kernels/.

More details on kernels can be found here.

Advanced

You can even go further with the configuration of your new kernel by updating the kernel.json. The content looks like this:

{
 "argv": [
     "/home/user/kernels/new-kernel/bin/python",
     "-m",
     "ipykernel_launcher",
     "-f",
     "{connection_file}"
 ],
 "display_name": "new-kernel",
 "language": "python"
}

It is possible to specify additional environment variables:

{
 "argv": [
     "/home/user/kernels/new-kernel/bin/python",
     "-m",
     "ipykernel_launcher",
     "-f",
     "{connection_file}"
 ],
 "display_name": "new-kernel",
 "language": "python",
 "env": {
     "variable": "value",
 }
}

Best practices

  • Where to install the new environment?

Based on the number and size of the python packages installed in your new enviroment, disk usage in your $HOME directory can easily exceed the limit. Therefore, it is preferable to create conda/virtualenv environments in /work (in the corresponding project). In that case, creating a conda env for example in work:

% conda  create --prefix /work/project_id/$USER ...
  • Helper script for kernel.json

You can create a start-kernel.sh shell script and make it executable (chmod +x start-kernel.sh). Inside the script you can put all configuration you want in your new kernel. An example can be for example loading system modules. The sctructure of the script can be like this:

#!/bin/bash

source /etc/profile
module purge

module load netcdf_c/4.3.2-gcc48
module load python/3.5.2
python -m ipykernel_launcher -f "$1"

And the kernel.json:

{
 "argv": [
     "start-kernel.sh",
     "{connection_file}"
 ],
 "display_name": "new-kernel",
 "language": "python"
}
  • uninstall/remove a kernel

    • jupyter kernelspec remove kernel-name

    • delete the corresponding conda/virtual environment if you don’t need it anymore

1

https://docs.jupyter.org/en/latest/projects/kernels.html#kernels-langs.