Working with Slurm

George G Vega Yon

2022-06-21

Introduction

Nowadays, high-performance-computing (HPC) clusters are commonly available tools for either in or out of cloud settings. Slurm Work Manager (formerly Simple Linux Utility for Resource Manager) is a program written in C that is used to efficiently manage resources in HPC clusters. The slurmR R package provides tools for using R in HPC settings that work with Slurm. It provides wrappers and functions that allow the user to seamlessly integrate their analysis pipeline with HPC clusters, putting emphasis on providing the user with a family of functions similar to those that the parallel R package provides.

Definitions

First, some important discussion points within the context of Slurm+R that users in general will find useful. Most of the points have to do with options available for Slurm, and in particular, with the sbatch command with is used to submit batch jobs to Slurm. Users who have used Slurm in the past may wish to skip this and continue reading the following section.

More information about Slurm can be found their official website here. A tutorial about how to use Slurm with R can be found here.

Submitting jobs via sbatch

In general, users will submit jobs to Slurm using the sbatch command line function. The sbatch function’s main argument is the name (path) to a bash script that holds the instructions (and sometimes options) associated to the program. Here is an example of an bash file to be submitted to Slurm

#!/bin/bash
#SBATCH --time=01:00:00
#SBATCH --job-name="A long job"
#SBATCH --mem=5GB
#SBATCH --output=long-job.out
cd /path/where/to/start/the/job

# This may vary per HPC system. At USC's hpc system
# we use: source /usr/usc/R/default/setup.sh
module load R

Rscript --vanilla long-job-rscript.R

This example bash file, which we name “long-job-rscript.slurm”, has the following components:

This batch script can be submitted to Slurm using the sbatch command line tool:

$ sbatch long-job-rscript.slurm

This is what happens under-the-hood in slurmR overall.


  1. For more on this see this thread on StackExchange.