Posted in 2020

Dask jobqueue on Mistral

According to the official Web site, Dask jobqueue can be used to deploy deploy Dask on job queuing systems like PBS, Slurm, MOAB, SGE, LSF, and HTCondor. Since the queuing system on Mistral is Slurm, we are going to show how to start a Dask cluster there. The idea is simple as described here. The difference is that the workers can be distributed through multiple nodes from the same partition. Using Dask jobqueue will Dask cluster as a Slurm jobs.

In this case, Jupyterhub will often play an interface role and the Dask can use more than the allocated resources to your jupyterhub session (profiles).

Jupyter notebook/lab extensions

Extensions bring additional interesting features to Jupyter*. Depending on the workflow in the notebook, users can install/enable extensions when required. Although is easy to add extensions to both Jupyter notebook an lab, the process can be sometimes annoying based on where jupyter is served from.

In general, installing and enabling extensions in your laptop or using the start-jupyter script is straightforward, especially when the developers well describe their extensions. There should be no restrictions or permissions issues, just follow the instructions.

Enable NCL Kernel in Jupyterhub

can’t use NCL (Python) as kernel in Jupyter

This tutorial won’t work

Single jupyter notebooks in containers

you are using singularity containers

you need jupyter notebooks

FileNotFoundError: [Errno 2] No such file or directory

you:

created your own conda env

Spawner options now savable

We introduced a new feature to the preset and advanced options form. This is a nice feature especially for the advanced options form, which contain many fields. You can also reset the options to their initial values by clicking on reset. The form options are saved in the client’s browser every 10 seconds and are not lost if:

the browser crashes

New Singularity module deployed

Recently, we deployed a new version of Singularity: 3.6.1. The old version is not available anymore due to many bugs reported by some users.

Errors like these are now fixed:

Connection error in your code?

Connection Error

DownloadWarning

VS Code Remote on Mistral

01 October 2020
Sofiane Bendoukha
containers

vs code is your favorite IDE

interested to use the remote extension

https://code.visualstudio.com/assets/docs/remote/remote-overview/architecture.png

Jupyterhub log file

Each Jupyter notebook is running as a SLUM job on MIstral. By default, stdout and stderr of the SLURM batch job that is spawned by Jupyterhub is written to your HOME directory on the HPC system. In order to make it simple to locate the log file:

if you use the preset options form: the log file is named jupyterhub_slurmspawner_preset_<id>.log.

GLIBC and the container-based workaround

Have you ever tried to install/use a software on Mistral and seen a message like this?

This is for example one of the reasons why PyTorch is not available in our python3 module. Those software packages require a newer version of glibc. Unfortunately, most of Mistral nodes are based on CentOS 6 kernel. To check the version of glibc:

Simple Dask clusters in Jupyterhub

There are multiple ways to create a dask cluster, the following is only an example. Please consult the official documentation. The Dask library is installed and can be found in any of the python3 kernels in jupyterhub. Of course, you can use your own python environment.

The simplest way to create a Dask cluster is to use the distributed module:

DKRZ Tech Talks

03 September 2020
Sofiane Bendoukha
news

It is our great pleasure to introduce the DKRZ Tech Talks. In this series of virtual talks we will present services of DKRZ and provide a forum for questions and answers. They will cover technical aspects of the use of our compute systems as well as procedures such as compute time applications and different teams relevant to DKRZ such as our machine learning specialists. The talks will be recorded and uploaded afterwards for further reference.

Go here for more information.