SLURM

Slurm is an open source, fault-tolerant, and highly scalable cluster management and job scheduling system for large and small Linux clusters.

It’s the job scheduler used by DKRZ.

Everytime an analysis is started from a plugin via website or scheduled through the command line the job is managed via SLURM. Here we will post some useful commands to better understand what is going on under the hood.

The status of the job can be consulted directly via:

$ squeue -u <username>

And can be cancel using the Job ID:

$ scancel <JobID>

We can even cancel all the jobs under our username alltogether:

$ scancel -u $USER

Note

only jobs launched under our username can be cancelled.

To get exhaustive info about all the nodes:

$ scontrol show node

...
NodeName=m20721 Arch=x86_64 CpuBind=cores CoresPerSocket=18
   CPUAlloc=72 CPUTot=72 CPULoad=42.09
   AvailableFeatures=64G
   ActiveFeatures=64G
   Gres=(null)
   NodeAddr=m20721 NodeHostName=m20721 Version=19.05.5-DKRZ.1
   OS=Linux 2.6.32-754.29.2.el6.x86_64 #1 SMP Thu May 7 06:14:05 EDT 2020
   RealMemory=64000 AllocMem=63360 FreeMem=51539 Sockets=2 Boards=1
   State=ALLOCATED ThreadsPerCore=2 TmpDisk=0 Weight=1 Owner=N/A MCS_label=N/A
   Partitions=compute2
   BootTime=2020-08-31T12:02:49 SlurmdStartTime=2020-08-31T12:03:53
   CfgTRES=cpu=72,mem=62.500G,billing=72
   AllocTRES=cpu=72,mem=63360M
   CapWatts=n/a
   CurrentWatts=0 AveWatts=0
   ExtSensorsJoules=n/s ExtSensorsWatts=0 ExtSensorsTemp=n/s
...

For a specific one:

$ scontrol show node <nodename>

To get some compact information about the nodes including node name and number of nodes, partition, node state, number of sockets, cores, threads, memory, disk and features:

$ sinfo --Node --long

Wed Sep 16 10:12:58 2020
NODELIST   NODES PARTITION       STATE CPUS    S:C:T MEMORY TMP_DISK WEIGHT AVAIL_FE REASON
m10000         1    shared   allocated   48   2:12:2  64000        0      1      64G none
m10001         1    shared   allocated   48   2:12:2  64000        0      1      64G none
m10002         1    shared   allocated   48   2:12:2  64000        0      1      64G none
m10003         1    shared   allocated   48   2:12:2  64000        0      1      64G none
...

where CPUS is the number of logical CPU cores available at each node:

CPU = CPU sockets (S) x physical cores per socket (C) x threads per physical cores (T)

In the example above each node has 48 logical CPUs: 2 CPU sockets with 12 physical cores each, each able to run 2 threads.

To know how many nodes and CPUs is taking each job (and who is running it):

$ squeue -o"%.7i %.9P %.8j %.8u %.2t %.10M %.6D %C"

  JOBID PARTITION     NAME     USER ST       TIME  NODES CPUS
2374336   prepost olgaxxx9  m300568 CG       5:38      1 4
2371188    shared kerberos  b302019 PD       0:00      1 1
2374276   compute  101_106  b309165 PD       0:00      1 1
2374276   compute  106_111  b309165 PD       0:00      1 1
2374277   compute  111_116  b309165 PD       0:00      1 1
2374277   compute  116_121  b309165 PD       0:00      1 1
2374277   compute  121_126  b309165 PD       0:00      1 1
2374277   compute  126_131  b309165 PD       0:00      1 1
2374277   compute  131_136  b309165 PD       0:00      1 1
...

More info about the format here.