How to set up Slurm-managed cronjobs#
To execute recurring batch jobs at specified dates, times, or intervals, you can use the Slurm scrontab tool. It provides a reliable alternative to the traditionally used cron utility to automate periodic tasks on Levante.
Set up and edit a crontab#
To define the recurring jobs, Slurm uses a configuration file,
so-called crontab, which is handled using the scrontab
command. The scrontab command with the -e option invokes an
editing session, so you can create or modify a crontab:
$ scrontab -e
The empty crontab contains only comment lines starting with # and
provides informations about the content, format and syntax of a
crontab:
#Welcome to scrontab, Slurm's cron-like interface.
#
# Edit this file to submit recurring jobs to be run by Slurm.
#
# Note that jobs will be run based on the Slurm controller's
# time and timezone.
# ...
# For example, the following line (when uncommented) would request
# a job be run at 5am each day.
# 0 5 * * * /my/script/to/run
#
# min hour day-of-month month day-of-week command
Per default the scrontab uses the editor set in the VISUAL
environment variable (or, if VISUAL is not set, EDITOR). You
can change it for the current editing session by assigning an
alternative value to the VISUAL variable, e.g.:
$ module load emacs
$ VISUAL=emacs scrontab -e
The #SCRON directives are used to define resources needed for the
job execution. These are essentially the same settings as for a regular
Slurm batch job, for example:
#SCRON --job-name=scrontab-example
#SCRON --ntasks=1
#SCRON --mem=5G
#SCRON --time=00:10:00
#SCRON --mail-type=FAIL
#SCRON --account=<prj-account>
#SCRON --output=/path/to/logs/scrontab-example-%j.log
#SCRON --partition=shared
#SCRON --oversubscribe
The #SCRON section is followed by a one-line crontab entry which
defines when and what should be executed. The scrontab syntax
for the date and time specifiers follows the classic
crontab(5)
syntax and can be looked up in the manual pages of the scrontab:
$ man scrontab
The crontab is automatically scheduled by the Slurm batch system after
saving and exiting the editor. You can see your cron-defined jobs with
the squeue command:
$ squeue --me
$ squeue --me -O Jobid,EligibleTime,Name,State
Hint
It is possible to have several crontab entries (jobs) in your
crontab. However, each entry needs its own #SCRON header.
Managing your scrontab jobs#
The content of the Slurm-managed crontab can be listed with the following command:
$ scrontab -l
If you want to cancel a job managed by scrontab, you need to
specify the --cron flag additionally:
$ squeue --me
$ scancel <jobid>
scancel: error: Kill job error on job id <jobid>: Cannot cancel scrontab jobs without --cron flag.
$ scancel --cron <jobid>
As a result, all lines belonging to this job in the crontab will be
preceded by the comment #DISABLED:
#DISABLED: #SCRON --job-name=scrontab-example
#DISABLED: #SCRON --ntasks=1
#DISABLED: #SCRON --mem=10G
The crontab entry can be enabled again by removing the
#DISABLED: comments.
All scrontab jobs have the same Slurm JobID. To append the new output
to the existing logfile instead of overwriting it, you can use the
--open-mode option, for example:
#SCRON --open-mode=append
Example#
The following example executes date && sleep 30 command every 10
minutes (*/10) and can be used as a template to get started.
#SCRON --job-name=scrontab-example
#SCRON --ntasks=1
#SCRON --time=00:02:00
#SCRON --mail-type=FAIL
#SCRON --account=<prj-account>
#SCRON --output=/path/to/logs/scrontab-example-%j.log
#SCRON --partition=shared
#SCRON --oversubscribe
#SCRON --open-mode=append
#SCRON --dependency=singleton
#min hour day-of-month month day-of-week command
*/10 * * * * date && sleep 30