Running Jobs with Slurm
Overview
All three clusters, Puma, Ocelote, and ElGato, use Slurm for resource management and job scheduling.
Additional Slurm Resources and Examples
Link | Description |
|---|---|
Official SchedMD user documentation. Includes detailed information on Slurm directives and commands. | |
Table for converting some common PBS job directives to Slurm syntax. | |
HPC Quick Start guide. If you have never submitted a batch job before, this is a great place to start. | |
Basic Slurm example scripts. Includes PBS scripts for comparison. | |
Growing repository of example Slurm submission scripts | |
A recorded video presentation of our Intro to HPC workshop. Keep your eyes peeled for periodic announcements in the HPC listserv on upcoming live sessions!= |
Node Summary
Before submitting a Slurm script, you must know (or at least have a general idea) of the resources needed for your job. This will tell you which type of node to request, how much memory, and other useful information that can be provided to the system via your batch script. A detailed list of Slurm batch flags are included below.
General Overview
Node Type | Description |
|---|---|
Standard CPU Node | This is the general purpose node, which can (and should) be used by the majority of jobs. |
High Memory CPU Node | Similar to the standard nodes, but with significantly more RAM. There a only a few of them and they should only be requested if you have tested your job on a standard node and find that its memory usage is too high. Both standard and high memory nodes share the same file system, so there is no advantage in terms of long term storage, only active RAM usage. |
GPU Node | Similar to the standard node, but with one or more GPUs available, depending on which cluster is in use. |
Hardware Limitations by Node Type and Cluster
Please consult the following table when crafting Slurm submission scripts. Requesting resources greater than what are available on a given cluster+node may lead to errors or delays.
Cluster | Node Type | N Nodes | N CPU/ Node | RAM/CPU | CPU RAM/ Node | N GPU/ Node | RAM/GPU | GPU RAM/ Node | Total N GPUs |
Puma | Standard | 236 | 94 | 5 gb | 470 gb | - | - | - | - |
High Mem | 3 standard | 94 | 32 gb | 3008 gb | - | - | - | - | |
2 buy-in | |||||||||
GPU | 8 standard | 94 | 5 gb | 470 gb | 4 | 32 gb | 128 gb | 32 | |
7 buy-in | 28 | ||||||||
Ocelote | Standard | 400 | 28 | 6 gb | 168 gb | - | - | - | - |
High Mem | 1 | 48 | 41 gb | 1968 gb | - | - | - | - | |
GPU | 46 | 28 | 8 gb | 224 gb | 1 | 16 gb | 16 gb | 46 | |
El Gato | Standard | 130 | 16 | 4 gb | 62 gb | - | - | - | - |
See here for example Slurm requests.
Other Job Limits
In addition to fitting your jobs within the constraints of our hardware, there are other limitations imposed by the scheduler to maintain fair use.
Time Limit Per Job: A single job cannot run for more than 10 days (240 hours). Requesting more time than this will lead to your job being stuck in the queue indefinitely with reason code "QOSMaxWallDurationPerJobLimit" (see below for more reason codes).
CPU Hours Per Group: The number of CPU hours used per job is subtracted from the PI's allocation. More info here.
Active jobs, CPUs, GPUs, and Memory: To see the limits and usage of these items, log onto the cluster you wish to know more about, and type "job-limits <group>"
Slurm and System Commands
Command | Purpose | Example(s) |
|---|---|---|
Native Slurm Commands | ||
| Submits a batch script for execution |
|
| Run parallel jobs. Can be in place of mpirun/mpiexec. Can be used interactively as well as in batch scripts |
|
| Requests a session to work on a compute node interactively | see: Interactive Sessions section below |
| Checks the status of pending and running jobs |
|
| Cancel a running or pending job |
|
| Place a hold on a job to prevent it from being executed |
|
| Releases a hold placed on a job allowing it to be executed |
|
System Commands | ||
| Displays your group membership, your account usage, and CPU allocation. Short for "view allocation" |
|
| Shortcut for quickly requesting an interactive job. Use "interactive --help" to get full usage. | interactive -a $GROUP_NAME |
| Retrieves a running or completed job's history in a user-friendly format |
|
| Retrieves a completed job's memory and CPU efficiency |
|
| Retrieves past jobs run by user. Can be used with option "-d N" to search for jobs run in the past N days. |
|
| View your group's job resource limits and current usage. |
|
| Display a visualization of nodes on a cluster and their usage |
|
| Display a text-based summary of a cluster's usage |
|
| Display a visualization of all three cluster's overall usage |
|
Batch Job Directives
Command | Purpose |
|---|---|
| Specify the account where hours are charged. Don't know your group name? Run the command "va" to see which groups you belong to |
| Set the job partition. This determines your job's priority and the hours charged. See Job Partition Requests below for additional information |
| Set the job's runtime limit in days, hours, minutes, and seconds. A single job cannot exceed 10 days or 240 hours. |
| Allocate N nodes to your job. For non-MPI enabled jobs, this should be set to "–-nodes=1" to ensure access to all requested resources and prevent memory errors. |
| ntasks specifies the number of tasks (or processes) the job will run. For MPI jobs, this is the number of MPI processes. Most of the time, you can use ntasks to specify the number of CPUs your job needs. However, in some odd cases you might run into issues. For example, see: Using Matlab By default, you will be allocated one CPU/task. This can be increased by including the additional directive --cpus-per-task. The number of CPUs a job is allocated is cpus/task * ntasks, or M*N |
| |
| Select N gb of memory per node. If "gb" is not included, this value defaults to MB. Directives --mem and --mem-per-cpu are mutually exclusive. |
| Select N GB of memory per CPU. Valid values can be found in the Node Types/Example Resource Requests section below. If "gb" is not included, this value defaults to MB. |
| Optional: (Ocelote and Puma only) Request N GPUs of any type. |
| Optional: (Puma only) Request N V100 GPUs. |
| Optional: (Puma only, Available 2/26/24) Request MIG GPU slice with 20 GB GPU memory. See MIG (Multi-Instance GPU) resources. |
| Optional: Request a high memory node (Ocelote and Puma only). |
| Submits an array job from indices N to M |
| Optional: Specify a name for your job. This will not automatically affect the output filename. |
| Optional: Specify output filename(s). If -e is missing, stdout and stderr will be combined. |
| Optional: Append your job's output to the specified output filename(s). |
| Optional: Request email notifications. Beware of mail bombing yourself. |
| Optional: Specify email address. If this is missing, notifications will go to your UArizona email address by default. |
| Optional: Request exclusive access to node. |
#SBATCH --export=VAR | Optional: Export a comma-delimited list of environment variables to a job. |
| Optional: Export your working environment to your job. |
| Optional: Do not export working environment to your job. |
Slurm Environment Variables
Variable | Purpose | Example Value |
|---|---|---|
| Job array's parent ID | 399124 |
| Total number of subjobs in the array | 4 |
| Job index number (unique for each job in the array) | 1 |
| Maximum index for the job array | 7 |
| Minimum index for the job array | 1 |
| Job array's index step size | 2 |
| Which cluster your job is running on | elgato |
| Points to the Slurm configuration file | /var/spool/slurm/d/conf-cache/slurm.conf |
| Number of CPUs allocated to target node | 3 |
| Number of GPUs allocated to the target node | 1 |
| Number of GPUs per node. Only set if --gpus-per-node is specified | 1 |
| Account being charged | groupname |
| The global GPU IDs of the GPUs allocated to the job. Only set in batch and interactive jobs. | 0 |
| Your Slurm Job ID | 399072 |
| Number of CPUs per node. This can be a list if there is more than one node allocated to the job. The list has the same order as SLURM_JOB_NODELIST | 3,1 |
| The job's name | interactive |
| The nodes that have been assigned to your job | gpu[73-74] |
| The number of nodes allocated to the job | 2 |
| The job's partition | standard |
| The job's QOS/Partition | qos_standard_part |
| The username of the person who submitted the job | netid |
| Same as SLURM_JOB_ID, your Slurm Job ID | 399072 |
| The memory/CPU ratio allocated to the job | 4096 |
| Same as SLURM_JOB_NUM_NODES – the number of nodes allocated to the job | 2 |
| Same as SLURM_JOB_NODELIST, The nodes that have been assigned to your job | gpu[73-74] |
| The number of tasks allocated to your job | 4 |
| Same as SLURM_NPROCS, the number of tasks allocated to your job | 4 |
| The directory where sbatch was used to submit the job | /home/u00/netid |
| The hostname where sbatch was used to submit the job | wentletrap.hpc.arizona.edu |
| The number of tasks to be initiated on each node. This can be a list if there is more than one node allocated to the job. The list has the same order as SLURM_JOB_NODELIST | 3,1 |
| Valid for interactive jobs, will be set with remote sibling cluster's IP address, port and RPC version so that any sruns will know which cluster to communicate with. | elgato:foo:0000:0000:000 |