Running Jobs with Slurm

# Running Jobs with Slurm

### Center for Advanced Research Computing <br> University of Southern California <br>

#### Last updated on 2026-06-23

---

## Outline

1. Slurm overview
2. Viewing node information
3. Running jobs
4. Monitoring jobs
5. Setting job dependencies
6. Running job arrays
7. Resources and support
8. Exercises

---

## Key terms and definitions

- **node** — individual computer
- **login node** — node for accessing cluster and submitting jobs
- **compute node** — node for computational workloads
- **compute resources** — CPUs, memory, GPUs, etc.
- **cluster** — set of interconnected compute nodes
- **partition** — distinct subset of compute nodes within a cluster
- **job** — allocation of compute resources assigned to a user for a period of time
- **scheduler** — application that controls when and where jobs are run on a cluster
- **terminal** — program for running shells with text input/output
- **shell** — program for processing commands and displaying their results
- **script** — text file with a sequence of commands to execute

---

## Section 1

### Slurm overview

---

## What is Slurm?

- Application for Linux cluster management and job scheduling
- Designed to be simple, scalable, portable, fault-tolerant, and interconnect-neutral
- Three main functions
  1. Allocates access to resources on compute nodes
  2. Provides a framework to run and monitor jobs on allocated compute nodes
  3. Manages a job queue for competing resource requests
- Configuration will differ across clusters
  - Typically configured with fairshare scheduling
  - Typically configured with job accounting database
- [Open source (GPL-2.0-or-later)](https://github.com/SchedMD/slurm)
- [Official docs](https://slurm.schedmd.com)

---

## What is a job?

- An allocation of compute resources assigned to a user for a period of time
  - CPUs, memory, GPUs, etc.
  - On one or more compute nodes
- Along with the computational tasks that run within the allocation
- Two types of jobs
  - Interactive job — for interactive usage (CLI, GUI)
  - Batch job — for batch scripts

---

## Overview of Slurm commands

| Category | Command | Purpose |
|---|---|---|
| Node info       | sinfo    | View compute partition and node information |
| Running jobs    | salloc   | Run an interactive job |
|                 | sbatch   | Run a batch job |
|                 | srun     | Launch tasks (job steps) within a job |
|                 | squeue   | View job queue information |
|                 | scancel  | Cancel pending or running jobs |
|                 | sprio    | View job priority information |
| Monitoring jobs | sacct    | View accounting information for jobs |
|                 | sstat    | View status information for running jobs |
|                 | sreport  | Generate reports from job accounting data |
| Other           | scontrol | View or modify Slurm configuration and state |

---

## Custom CARC Slurm commands

| Command | Purpose |
|---|---|
| myaccount | View account information for user |
| noderes   | View available resources on nodes |
| jobqueue  | View job queue information |
| jobhist   | View compact history of jobs |
| jobinfo   | View detailed job information |
| jobeff    | View job efficiency information |

---

## Section 2

### Viewing node information

---

## sinfo

- View compute partition and node information
- Default output lists information by partition and node status
- `sinfo --help`
- [https://slurm.schedmd.com/sinfo.html](https://slurm.schedmd.com/sinfo.html)

```bash
$ sinfo

$ sinfo -lNp debug
```

---

## sinfo notes

- Useful options include `--Node (-N)`, `--partition (-p)`, and `--states`
- Many formatting options with `--format (-o)` or `--Format (-O)`
  - Can also use SINFO_FORMAT environment variable to set
  - `export SINFO_FORMAT=...`
  - Add to `~/.bashrc` to automatically set when logging in

---

## Codes for common node states

- allocated
  - The node has been allocated to one or more jobs
- down
  - The node is unavailable for use
- draining
  - The node is currently executing a job, but will not be allocated additional jobs
- idle
  - The node is not allocated to any jobs and is available for use
- maint
  - The node is currently in a reservation with a flag value of "maintenance"
- mixed
  - The node has some of its CPUs allocated while others are idle
- reserved
  - The node is in an advanced reservation and not generally available
- planned
  - The node is planned for a job

---

## noderes

- View available resources (free or configured) on nodes
  - CPUs
  - Memory
  - GPUs
- Convenience command based on output from `scontrol show node`
- Information that `sinfo` does not provide
- Output formatted as table with one row per node
- `noderes --help`

```bash
$ noderes

$ noderes -c -p largemem

$ noderes -g

$ noderes -f -g
```

---

## Discovery cluster partitions

| Partition | Purpose | Timelimit |
|---|---|---|
| main       | Serial and small-to-large parallel jobs | 2 days |
| gpu        | Jobs requiring GPUs | 2 days |
| largemem   | Jobs requiring large amounts of memory (up to 1.5 TB) | 7 days |
| debug      | Short-running jobs for testing or debugging purposes | 1 hour |
| oneweek    | Long-running jobs | 7 days |

---

## Section 3

### Running jobs

---

## Commands for running jobs

- `salloc` — run an interactive job
- `sbatch` — run a batch job
- `srun` — run tasks (job steps) within a job

**Do not run computational tasks on login nodes!**

---

## salloc

- Run an interactive job
- How it works
  - Submits allocation request to scheduler
  - Scheduler plans and allocates resources on one or more compute nodes
  - Then launches a shell on one of the allocated compute nodes
  - User can then run commands interactively

```bash
$ salloc

$ salloc -p debug -c 4 --mem=8G
```

---

## salloc example

Request 8 CPU cores and 16 GB of memory:

```bash
$ salloc -c 8 --mem=16G
salloc: Granted job allocation 9156692
salloc: Nodes b11-09 are ready for job
```

Once allocated, then run commands interactively.

For example, load and run R:

```bash
$ module purge
$ module load r/4.6.0
$ R --version
```

When done, exit back to the login node:

```bash
$ exit
exit
salloc: Relinquishing job allocation 9156692
salloc: Job allocation 9156692 has been revoked.
```

---

## salloc notes

- Useful for developing code, troubleshooting, and debugging
- Interactive usage will necessarily waste compute resources
  - Idle resources that others cannot use
  - So try not to request too much and exit when done
- Similar options to `sbatch`
- `salloc --help`
- [https://slurm.schedmd.com/salloc.html](https://slurm.schedmd.com/salloc.html)

---

## Interactive jobs with GUI apps

- To run a GUI app within a job, use [CARC OnDemand](https://ondemand.carc.usc.edu)
  - Traveler Desktop (MATLAB, Stata, etc.)
  - Code Server
  - JupyterLab
  - RStudio Server
- These apps run as Slurm jobs on compute nodes

---

## sbatch

- Run a batch job
- Batch mode means there is no user input during the job
- How it works
  - Submits allocation request to scheduler with job script
  - Scheduler plans and allocates resources on one or more compute nodes
  - Then runs the submitted job script on one of the allocated compute nodes
  - Output messages are saved to a log file in the submission directory

```bash
$ sbatch sim.slurm

$ sbatch -p debug sim.slurm
```

---

## General structure of a job script

1. Specify shell script hashbang (e.g., `#!/bin/bash`)
2. Specify `sbatch` options (e.g., resource requests)
3. Load software as needed (e.g., modules)
4. Modify environment as needed (e.g., set environment variables)
5. Run program or script (e.g., `julia script.jl`)

```bash
#!/bin/bash

#SBATCH --account=<project_id>
#SBATCH --partition=main
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=1
#SBATCH --mem=2G
#SBATCH --time=1:00:00

module purge
module load julia/1.12.6

export JULIA_DEPOT_PATH=/project2/ttrojan_123/julia/depot

julia script.jl
```

---

## Example for serial job (single-core)

Estimating pi with Julia using a single CPU core:

- [Slurm job script](https://github.com/uschpc/workshop-slurm/blob/main/examples/pi-serial.slurm)
  - [Julia script](https://github.com/uschpc/workshop-slurm/blob/main/examples/pi.jl)

Submit job using `sbatch`:

```bash
$ sbatch pi-serial.slurm
Submitted batch job 956553
```

View log file using `less`:

```bash
less slurm-956553.out
```

Press `q` to exit `less`

---

## Job log files

- Job output messages are saved to a log file
- File is first created and then updated as the job progresses
- Default settings for log file
  - Named `slurm-<job_id>.out`
  - Saved to the submission directory
  - Both output and error messages are added to the same file
- Use `--output` and/or `--error` options to customize filename or split output

---

## Typical batch job workflow

1. Create job script
2. Submit job script with `sbatch`
3. Check job status with `myqueue` or `jobhist --me`
4. When job completes, check log file
5. If job failed, modify job script and resubmit (back to step 1)
  - Look for warning and error messages in log file
  - Check detailed job information with `jobinfo`
6. If job succeeded, check detailed job information with `jobinfo`
  - Review efficiency values (time, CPU, memory)
  - If feasible, use less resources next time for a similar job

---

## Example for parallel job (single node)

Estimating pi with Julia using multiple CPU cores:

- [Slurm job script](https://github.com/uschpc/workshop-slurm/blob/main/examples/pi-parallel.slurm)
  - [Julia script](https://github.com/uschpc/workshop-slurm/blob/main/examples/pi.jl)

Submit job using `sbatch`:

```bash
$ sbatch pi-parallel.slurm
Submitted batch job 956554
```

---

## Example for parallel job (multiple nodes)

Running HPCG benchmark with MPI using multiple CPUs across multiple nodes:

- [Slurm job script](https://github.com/uschpc/workshop-slurm/blob/main/examples/hpcg.slurm)

Submit job using `sbatch`:

```bash
$ sbatch hpcg.slurm
Submitted batch job 956555
```

---

## Example for GPU job

Getting GPU information:

- [Slurm job script](https://github.com/uschpc/workshop-slurm/blob/main/examples/gpu.slurm)

Submit job using `sbatch`:

```bash
$ sbatch gpu.slurm
Submitted batch job 956556
```

---

## GPU notes

View available GPU models:

```bash
$ noderes -z -g
```

To request any GPU model:

```bash
#SBATCH --gpus-per-task=1
```

To request a specific GPU model (e.g., A100):

```bash
#SBATCH --gpus-per-task=a100:1
```

To request a GPU model from a subset (e.g., A100 or A40):

```bash
#SBATCH --constraint="a100|a40"
#SBATCH --gpus-per-task=1
```

---

## sbatch notes

- Can also submit batch jobs via [CARC OnDemand](https://ondemand.carc.usc.edu) using [Project Manager app](https://ondemand.carc.usc.edu/pun/sys/dashboard/projects)
- Slurm copies the job script
  - So modifying it after submitting will not impact the job
  - But modifying other job files (data, scripts, etc.) will impact the job
- By default, shell environment is transferred to job when submitted
- Similar options to `salloc`
- Command-line option will override option given in job script
- `sbatch --help`
- [https://slurm.schedmd.com/sbatch.html](https://slurm.schedmd.com/sbatch.html)

---

## Commonly used salloc/sbatch options

| Option | Description | Default value |
|---|---|---|
| `--account=<project_id>`       | Account to charge resources to | default |
| `--partition=<partition_name>` | Request nodes on specified partition | main |
| `--constraint=<attribute>`     | Node feature to request (e.g., `epyc-7513`) | n/a |
| `--nodes=<number>`             | Number of nodes to use | 1 |
| `--ntasks=<number>`            | Number of processes to run | 1 |
| `--cpus-per-task=<number>`     | Number of CPUs (cores) per task | 1 |
| `--mem=<number>`               | Memory per node | n/a |
| `--mem-per-cpu=<number>`       | Memory per CPU (core) | 2G |
| `--time=<D-HH:MM:SS>`          | Maximum run time | 1:00:00 |
| `--mail-type=<value>`          | Email notifications (e.g., `all`) | n/a |
| `--mail-user=<address>`        | Email address | n/a |

---

## Job limits

- Various limits are applied to jobs (to promote fairshare and efficiency)
- Limits may vary by partition or change over time
- Common limits applied to jobs
  - Maximum run time
  - Maximum concurrent CPUs
  - Maximum concurrent memory
  - Maximum concurrent GPUs
  - Maximum number of jobs queued
  - Maximum number of jobs running

```bash
$ sinfo -s

$ sacctmgr show qos format=name,maxtrespu%70,maxjobspu,maxsubmitpu
```

---

## Slurm environment variables

- Output variables can be used in job or application scripts
- [https://slurm.schedmd.com/sbatch.html#SECTION_OUTPUT-ENVIRONMENT-VARIABLES](https://slurm.schedmd.com/sbatch.html#SECTION_OUTPUT-ENVIRONMENT-VARIABLES)

| Variable | Description |
|---|---|
| SLURM_JOB_ID         | The ID of the job allocation |
| SLURM_JOB_NODELIST   | List of nodes allocated to the job |
| SLURM_JOB_NUM_NODES  | Total number of nodes in the job's resource allocation |
| SLURM_NTASKS         | Number of tasks requested |
| SLURM_CPUS_PER_TASK  | Number of CPUs requested per task |
| SLURM_SUBMIT_DIR     | The directory in which `sbatch` was invoked |
| SLURM_ARRAY_TASK_ID  | Job array ID (index) number |

---

## srun

- Launch tasks (job steps) within a job
- Primarily used for launching parallel tasks for MPI programs
  - Alternative to `mpirun` launcher
  - May launch tasks faster than `mpirun`
- `srun --help`
- [https://slurm.schedmd.com/srun.html](https://slurm.schedmd.com/srun.html)

---

## Related commands

- `squeue` — view job queue information
- `scancel` — cancel pending or running jobs
- `sprio` — view job priority information

---

## squeue

- View job queue information
- Use `jobqueue` for better output format
- `squeue --help`
- [https://slurm.schedmd.com/squeue.html](https://slurm.schedmd.com/squeue.html)
- [https://slurm.schedmd.com/job_reason_codes.html](https://slurm.schedmd.com/job_reason_codes.html)

```bash
$ squeue

$ squeue --me

$ jobqueue --me

$ jobqueue -p largemem
```

---

## Common codes for job state

- PD / PENDING
  - Job is awaiting resource allocation
- R / RUNNING
  - Job currently has an allocation
- CG / COMPLETING
  - Job is in the process of completing. Some processes on some nodes may still be active
- CD / COMPLETED
  - Job has terminated all processes on all nodes with an exit code of zero
- CA / CANCELLED
  - Job was explicitly canceled by the user or system administrator. The job may or may not have been initiated

---

## Common codes for PD / PENDING reason

- Resources
  - The job is waiting for resources to become available
- Priority
  - One or more higher priority jobs exist for this partition or advanced reservation
- ReqNodeNotAvail
  - Some node specifically required by the job is not currently available
- QOSMaxJobsPerUserLimit
  - The job has reached the maximum jobs per user limit
- QOSMaxCpuPerUserLimit
  - The job has reached the maximum CPU per user limit
- QOSMaxGRESPerUserLimit
  - The job has reached the maximum GRES (e.g., GPU) per user limit
- AssocGrpCPUMinutesLimit
  - The project account has run out of CPU time
- InvalidAccount
  - The project account is invalid

---

## squeue notes

- Completed jobs will not show up in `squeue` output
- Useful options include `--start` and `--partition`
- Many formatting options with `--format` or `--Format`
  - Can also use SQUEUE_FORMAT environment variable to set
  - `export SQUEUE_FORMAT=...`
  - Add to `~/.bashrc` to automatically set when logging in
- Create shorter alias
  - `alias myq="squeue --me"`
  - Or `alias myq="jobqueue --me"`
  - Add to `~/.bashrc` to automatically set when logging in

---

## scancel

- Cancel pending or running jobs
- Only works for your own jobs
- Use specific job ID or cancel all your jobs
- `scancel --help`
- [https://slurm.schedmd.com/scancel.html](https://slurm.schedmd.com/scancel.html)

```bash
$ scancel <job_id>

$ scancel -u $USER
```

---

## Job priorities

- Jobs are assigned priorities by the scheduler
- Priority values depend on a number of factors
  - Fairshare (amount of resources recently used by user)
  - Partition requested for job
  - Age of job
  - Size of job
- Each factor has an associated weight (depends on Slurm configuration)
- [https://slurm.schedmd.com/priority_multifactor.html](https://slurm.schedmd.com/priority_multifactor.html)
- [https://slurm.schedmd.com/fair_tree.html](https://slurm.schedmd.com/fair_tree.html)

---

## sprio

- View job priority information (for pending jobs)
- Output can be difficult to interpret
- If normalized, a priority value closer to 1 means a higher priority
- `sprio --help`
- [https://slurm.schedmd.com/sprio.html](https://slurm.schedmd.com/sprio.html)

```bash
$ sprio -nj <job_id>

$ sprio -nu $USER

$ sprio -nlp gpu
```

---

## Section 4

### Monitoring jobs

---

## Types of job monitoring

- Different types of monitoring based on level
  - Workflow (status of workflow)
  - Job (status of job)
  - Node (utilization of resources)
- Monitor resource utilization by node within job
  - CPU utilization
  - Memory utilization
  - GPU utilization

---

## Commands for monitoring jobs

- `sacct` — view accounting information for jobs
- `sstat` — view status information for running jobs
- `jobhist` — view compact history of jobs
- `jobinfo` — view detailed job information

---

## sacct

- View accounting information for jobs
- Including resource allocations and utilization
- `sacct --help`
- [https://slurm.schedmd.com/sacct.html](https://slurm.schedmd.com/sacct.html)
- [https://slurm.schedmd.com/job_state_codes.html](https://slurm.schedmd.com/job_state_codes.html)

```bash
$ sacct

$ sacct -j <job_id>

$ sacct --format=JobID,MaxRSS,AveCPUFreq,MaxDiskRead,MaxDiskWrite,State,ExitCode
```

---

## sacct notes

- By default, only lists jobs from past day
- Some useful options are `--starttime`, `--endtime`, `--brief`, and `--state`
- Many formatting options with `--format`
  - Can also use SACCT_FORMAT environment variable to set
  - `export SACCT_FORMAT=...`
  - Add to `~/.bashrc` to automatically set when logging in

---

## Job exit codes

- 0 indicates success
- Non-zero indicates failure
- Exit code 1 indicates a general failure
- Exit code 2 indicates incorrect use of shell builtins
- Exit codes 3-124 indicate some error in job (check software exit codes)
- Exit code 125 indicates out of memory
- Exit code 126 indicates command cannot execute
- Exit code 127 indicates command not found
- Exit code 128 indicates invalid argument to exit
- Exit codes 129-192 indicate jobs terminated by Linux signals
  - For these, subtract 128 from the number and match to signal code
  - Run `kill -l` to list signal codes
  - Run `man signal` for more information

---

## sstat

- View status information for running jobs
- Primarily used to monitor resource utilization
- `sstat --help`
- [https://slurm.schedmd.com/sstat.html](https://slurm.schedmd.com/sstat.html)

```bash
$ sstat -j <job_id>

$ sstat -j <job_id> --format=JobID,AveCPUFreq,MaxRSS,MaxDiskRead,MaxDiskWrite
```

---

## jobhist

- View compact history of jobs
- Convenience command with better output format compared to `sacct`
- `jobhist --help`

```bash
$ jobhist --me

$ jobhist --me -s 2026-05-25

$ jobhist -p largemem
```

---

## jobinfo

- View detailed job information
- Convenience command with better output format compared to `sacct` or `sstat`
- Also includes efficiency information not provided by `sacct` or `sstat`
- `jobinfo --help`

```bash
$ jobinfo <job_id>

$ jobinfo $SLURM_JOB_ID
```

---

## jobeff

- View job efficiency information
- Includes CPU and memory efficiency
- Useful for reviewing efficiency information for many jobs
- `jobeff --help`

```bash
$ jobeff --me

$ jobeff --me -s 2026-05-25
```

---

## sreport

- Generate reports from job accounting data
- `sreport --help`
- [https://slurm.schedmd.com/sreport.html](https://slurm.schedmd.com/sreport.html)

```bash
$ sreport cluster accountutilizationbyuser start=2023-04-01 end=2023-04-30 users=$USER

$ sreport user topusage start=2023-04-01 end=2023-04-30 accounts=<project_id>
```

---

## scontrol

- View or modify Slurm configuration and state
- Primarily for admins, but some subcommands can be useful for users
- `scontrol --help`
- [https://slurm.schedmd.com/scontrol.html](https://slurm.schedmd.com/scontrol.html)

```bash
$ scontrol show partition <partition>

$ scontrol show node <node_id>

$ scontrol show job <job_id>

$ scontrol hold <job_id>

$ scontrol release <job_id>
```

---

## Section 5

### Setting job dependencies

---

## When a job depends on other jobs

- Defer the start of a job until the specified dependencies have been satisfied
- Some example use cases
  - Separating data processing and analysis into multiple jobs
  - Checkpointing and restarting long-running jobs
- Setting a job dependency
  - Need job IDs for relevant jobs (use `squeue --me` if needed)
  - Different dependency types available
  - Add `#SBATCH --dependency=<type>` option to job script
  - Or use `sbatch --dependency=<type>` command-line option
- [https://slurm.schedmd.com/sbatch.html#OPT_dependency](https://slurm.schedmd.com/sbatch.html#OPT_dependency)

---

## Commonly used job dependency types

`--dependency=afterok:<job_id>[:<job_id>...]`

- This job can begin execution after the specified jobs have successfully executed (ran to completion with an exit code of 0)

`--dependency=afternotok:<job_id>[:<job_id>...]`

- This job can begin execution after the specified jobs have terminated in some failed state (non-0 exit code, node failure, timed out, etc.)

`--dependency=after:<job_id>[[+time][:<job_id>[+time]...]]`

- After the specified jobs start or are cancelled and time in minutes from job start or cancellation happens, this job can begin execution. If no time is given, then there is no delay after start or cancellation

`--dependency=afterany:<job_id>[:<job_id>...]`

- This job can begin execution after the specified jobs have terminated

---

## Example for job dependency

```bash
$ sbatch jl.slurm
Submitted batch job 3353548

$ sbatch --dependency=afterok:3353548 jl2.slurm
Submitted batch job 3353549

$ squeue --me
  JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
3353549      main jl2.slur  ttrojan PD       0:00      1 (Dependency)
3353548      main jl.slurm  ttrojan  R       0:32      1 b22-31
```

---

## Section 6

### Running job arrays

---

## Job arrays

- For submitting and managing collections of similar jobs quickly and easily
- Create and submit one job script to run many jobs
- Some example use cases
  - Varying simulation or model parameters
  - Running the same file processing steps on different files
- Setting up a job array
  - Add `#SBATCH --array=<index>` option to job script
  - Then modify job or application script to use array index
  - Typically via `$SLURM_ARRAY_TASK_ID` variable
- Each job in the array is independent and has the same resource requests
- [https://slurm.schedmd.com/job_array.html](https://slurm.schedmd.com/job_array.html)
- For lots of short-running jobs, pack them in a single Slurm job using [HyperShell](https://hypershell.readthedocs.io)

---

## Example for job array

Simple hello world:

- [Slurm job script](https://github.com/uschpc/workshop-slurm/blob/main/examples/array.slurm)

Submit job array using `sbatch`:

```bash
$ sbatch array.slurm
Submitted batch job 3355483
```

Check job queue to see multiple jobs:

```bash
$ squeue --me
    JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
3355483_1      main array.sl  ttrojan  R       0:05      1 d05-35
3355483_2      main array.sl  ttrojan  R       0:05      1 e16-15
3355483_3      main array.sl  ttrojan  R       0:05      1 d18-29
```

Check job log files to see different output

---

## Section 7

### Resources and support

---

## Additional resources

- [Slurm documentation](https://slurm.schedmd.com)
- [CARC Slurm cheatsheet](https://www.carc.usc.edu/user-guides/hpc-systems/using-our-hpc-systems/slurm-cheatsheet)
- [CARC Slurm CLI tools](https://github.com/uschpc/slurm-tools)

---

## CARC support

- [Workshop materials](https://github.com/uschpc/workshop-slurm)
- [Submit a support ticket](https://www.carc.usc.edu/user-support/submit-ticket)
- Office Hours
  - Every Tuesday 2:30-5pm
  - Get Zoom link [here](https://www.carc.usc.edu/user-support/office-hours-and-consultations)

---

## Section 8

### Exercises

---

## Exercises

1. View configured and free resources for nodes in the largemem partition
2. Start and end an interactive job on the main partition
3. Submit a "Hello world" batch job to the debug partition
4. Check efficiency of your jobs from the past week