Single jupyter notebooks in containers#

Why?#

  • you are using singularity containers

  • you need jupyter notebooks

What we already have?#

For single jupyter notebooks on Mistral, we have the start-jupyter script. This script makes it possible starting jupyter notebooks by providing some parameters and without strugling with ssh tunneling issues.

% ./start-jupyter --help
Usage: start-jupyter [OPTION]

Available values for OPTION include:

  -A acctcode    start a job with acctcode
  -c command     invoke jupyter with command
  -d             check for presence of jupyter
  -i file        source file prior to running jupyter
  -n ntasks      request ntasks tasks for job
  -p partition   run job on partition
  -t time        job runtime
  -u username    use username for login

What is missing?#

The script does not support singularity, it just looks for the jupyter binary in the Mistral host and use it to start the notebook. In order to use another path (e.g. from a container), this must be specified. So, the idea is simple: instead of using the remote host jupyter, I will use jupyter from the singularity container. This page explains how to use Singularity containers on Mistral.

The solution#

I modified the original script to support singularity containers. Now, if you specify a singularity image with the -s flag, the script will use it to start the notebook.

% ./start-jupyter --help
Usage: start-jupyter [OPTION]

Available values for OPTION include:

  -A acctcode    start a job with acctcode
  ...
  -u username    use username for login
  -s image       singularity image

The script needs the Singularity module loaded, so I just use the jupyter_preload file for this purpose.

module load python3/unstable
module load singularity
All I need is a working image with jupyter installed, so I did create it and push it to the hub.
The image must be available in the HOME directory.

This is how I call the script:

./start-jupyter -u xxxxxxx -i jupyter_preload -s jupyter_mistral_latest.sif

The new script#

The modified script can be found here. These modifications can be merged with the original script later.

Next steps#

  • download/pull the image if not available

  • create further customized images

Note

Depending on your workflow and the required softwares/packages, additional steps might be necessary, thus the jupyter_preload needs to be adapted accordingly.

FileNotFoundError: [Errno 2] No such file or directory   Enable NCL Kernel in Jupyterhub