Using GPU nodes#
Overview#
An increasing number of scientific software relevant to earth system
research can take advantage of GPGPU (General Purpose
computing on Graphics Processing Units)
technology. Therefore, a considerable part of the Levante cluster
consists of nodes equipped with GPUs. Such nodes make up the Slurm
partition named gpu
, accordingly. They are intended for the
following use cases:
Production runs with GPU-ready model codes
Porting and development for GPUs of model codes not yet adapted
Using of ML methods for data processing and analysis
3D visualization benefiting from hardware acceleration
All Levante GPU nodes are equipped with 4 GPUs of type NVIDIA A100-SXM4 (Ampere architecture). Four nodes from Levante phase one have 40GB memory per GPU, the rest has 80GB. The table below gives an overview of the node configuration.
GPUs |
CPU arrangement |
#Nodes |
GPU type tag/ feature |
---|---|---|---|
4x Nvidia A100 40GB/GPU |
2x AMD 7763 CPU; 128 cores in total, 512 GB/1024 GB main memory |
2/2 |
|
4x Nvidia A100 80GB/GPU |
2x AMD 7763 CPU; 128 cores in total, 512 GB main memory |
56 |
|
To inquire more details about GPU hardware, you can allocate a GPU node with:
$ salloc -p gpu --gpus=4 -A <prj-account>
and execute the following commands:
$ nvidia-smi -q
$ module load nvhpc/22.5-gcc-11.2.0
$ nvaccelinfo
The GPU partition#
Levante’s GPU nodes can be accessed via the SLURM parition gpu
:
#SBATCH --partition=gpu
Note
Only projects with an allocation for the GPU resource can submit
jobs to the gpu
partition. Read more about requesting
the GPU resource.
Individual GPUs of nodes can be requested separately, therefore you must specify the number of GPUs needed explicitly, using the --gpus
option:
#SBATCH --gpus=1
There are various ways to specify the GPU resources for your job, which are documented in the Slurm documentation. We recommend to use one of the following:
Slurm option |
Comment |
---|---|
|
Number of GPUs per job |
|
Number of GPUs per node |
|
number of tasks must be known |
Here, count
and type
have to be replaced accordingly.
The optional argument type
can be any of the GPU type tag listed above.
However, we observed unexpected behavior when using this option, therefore we recommend to skip it.
In case the exact GPU type matters for your job, better use --constraint=
and specify the nodes’ feature.
The GPU nodes’ features are listed in the table at the beginning of this page.
When no GPU type is specified, Slurm will assign the hardware available first to your job.
To start a job with two tasks and one GPU per task you could for example define it as follows:
#SBATCH --ntasks=2
#SBATCH --gpus-per-task=1
If all 4 GPUs of a node are to be used, the most flexible way to make all
devices visible is to use the --exclusive
option:
#SBATCH --exclusive
With this method, more complicated setups (e.g. 5 MPI tasks of which only 4 use GPUs) can be handled.
Basic example Slurm scripts using GPUs can be found in Example Batch Scripts.