Basic Array JobArray jobs are used to execute the same script multiple times with different input. What problem does this help fix? To execute multiple analyses, a user may be tempted to submit jobs with a scripted loop, e.g.: Code Block |
---|
language | bash |
---|
theme | Midnight |
---|
| for i in $( seq 1 10 ); do sbatch script.slurm <submission options> ; done |
This isn’t a good solution because it submits too many jobs too quickly and overloads the scheduler. Instead, an array job can be used to achieve the same ends. Example Code Block |
---|
language | bash |
---|
theme | Midnight |
---|
| #!/bin/bash
#SBATCH --ntasks=1
#SBATCH --nodes=1
#SBATCH --time=00:01:00
#SBATCH --partition=standard
#SBATCH --account=YOUR_GROUP
#SBATCH --array 1-5
echo "./sample_command input_file_${SLURM_ARRAY_TASK_ID}.in" |
Script Breakdown What differentiates the script above from standard submissions is the --array directive. This is what tells SLURM that you’re submitting an array. Following this flag, you will specify the number of jobs you wish to run. In this case, we’re running 5: Code Block |
---|
language | bash |
---|
theme | Midnight |
---|
| #SBATCH --array 1-5 |
Each job in the array has its own associated environment variable SLURM_ARRARY_TASK_ID that can be used to differentiate subjobs. To demonstrate how we can use each of these to read in different input files, we’ll print a sample command: Code Block |
---|
language | bash |
---|
theme | Midnight |
---|
| echo "./sample_command input_file_${SLURM_ARRAY_TASK_ID}.in" |
Script Submission Code Block |
---|
language | bash |
---|
theme | Midnight |
---|
| (ocelote) [netid@junonia ~]$ sbatch basic_array_job.slurm
Submitted batch job 73958 |
Output FilesEach of the subjobs in the array will produce its own output file of the form slurm_jobid_arrayid.out as seen below: Code Block |
---|
language | bash |
---|
theme | Midnight |
---|
| echo "./sample_command input_file_${SLURM_ARRAY_TASK_ID}.in" |
For more information on naming SLURM files, see our online documentation. File ContentsBelow is a concatenation of the job’s output files. Notice how the array indices function to differentiate the input files in the sample command: Code Block |
---|
language | bash |
---|
theme | Midnight |
---|
| (ocelote) [netid@junonia ~]$ cat slurm-73958_* | grep sample
./sample_command input_file_1.in
./sample_command input_file_2.in
./sample_command input_file_3.in
./sample_command input_file_4.in
./sample_command input_file_5.in |
|