GLIBC and the container-based workaround¶
Have you ever tried to install/use a software on Mistral and seen a message like this?
/lib64/libc.so.6: version `GLIBC_2.14` not found
This is for example one of the reasons why PyTorch is not available
in our python3 module. Those software packages require a newer version
glibc. Unfortunately, most of Mistral nodes are based on CentOS 6
kernel. To check the version of
You will get an output like this:
ldd (GNU libc) 2.12 Copyright (C) 2010 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. Written by Roland McGrath and Ulrich Drepper.
So, how can you in some cases overcome this issue?
For many of the issues with
glibc, we propose the use of
singularity containers. The advantage is that you can use different
Operating Systems running a higher version of
glibc per default.
image with pytorch installed (you can also create it and then push it to Mistral)
os with a kernel supported on Mistral
The PyTorch case
Let’s take the simplest way:
check if it works on Mistral (Kernel)
load singularity module on MIstral
pull the image
run/execute the container
use your software
Step (1) can be more complicated depending on your workflow and which other softwares are required: gpu, cuda, cudnn, nvidia drivers.
Here a concrete example:
% module load singularity/3.6.1-gcc-9.1.0 % singularity pull --name pytorch_gpu.simg shub://yinglilu/pytorch_gpu_singularity:1.3.0 INFO: Use cached image % singularity shell pytorch_gpu.simg Singularity> python Python 3.7.3 (default, Mar 27 2019, 22:11:17) [GCC 7.3.0] :: Anaconda custom (64-bit) on linux Type "help", "copyright", "credits" or "license" for more information. >>> import torch >>>
If you check the version of
glibc inside the container, you will
notice it has changed to a higher version.
Singularity> ldd –version ldd (Debian GLIBC 2.24-11+deb9u4) 2.24 Copyright (C) 2016 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. Written by Roland McGrath and Ulrich Drepper. Singularity> uname -r 2.6.32-754.29.2.el6.x86_64 Singularity> cat /etc/os-release PRETTY_NAME=“Debian GNU/Linux 9 (stretch)” NAME=“Debian GNU/Linux” VERSION_ID=“9” VERSION=“9 (stretch)” ID=debian HOME_URL=“https://www.debian.org/” SUPPORT_URL=“https://www.debian.org/support” BUG_REPORT_URL=“https://bugs.
The Caffe framework case
Another kernel related issue occurred when trying to build the [Caffe deep learning framework] on Mistral. Hopefully, they have some Docker images which are supported on Mistral (after conversion to singularity).
Example with the cpu version:
% module load singularity/3.6.1-gcc-9.1.0 % singlularity pull docker://bvic/caffe:cpu % singularity run caffe_cpu.sif
“Assumption” In this article, I assume that you are already familiar with the concept of container images and virtualization.
Please take a look here for more information on how to use singularity on Mistral.
Currently, containers are seen as workaround for some issues on Mistral. So, do not expect they will solve all problems.