Difference between revisions of "LSF to Slurm quick reference"

Latest revision as of 16:55, 23 February 2023

Introduction

The commands for Slurm are similar to the ones used in LSF. You can find a mapping of the relevant commands below.

Job submission

Simple command

Frequently used bsub/sbatch options

Shell script

#!/bin/bash

#BSUB -n 4
#BSUB -W 08:00
#BSUB -R "rusage[mem=2000]"
#BSUB -R "rusage[scratch=1000]"    # per core
#BSUB -J analysis1
#BSUB -o analysis1.out
#BSUB -e analysis1.err

module load xyz/123
command1
command2
...

#!/bin/bash

#SBATCH -n 4
#SBATCH --time=8:00:00
#SBATCH --mem-per-cpu=2000
#SBATCH --tmp=4000                        # per node!!
#SBATCH --job-name=analysis1
#SBATCH --output=analysis1.out
#SBATCH --error=analysis1.err

module load xyz/123
command1
command2
...

Note:

• In LSF, the jobscript.sh must be passed to bsub via the "<" operator
• In LSF, scratch space is expressed per core, while in Slurm it is per node
• In LSF, the default output file is "lsf.oJOBID", while in Slurm it is "slurm-JOBID.out"

Interactive job

Parallel job

Shared memory (OpenMP, threads)

Distributed memory (MPI, processes)

The Slurm options

• --ntasks-per-core,
• --cpus-per-task,
• --nodes, and
• --ntasks-per-node

are supported.

Please note that for larger parallel MPI jobs that use more than a single node (more than 128 cores), you should add the sbatch option

-C ib

to make sure that they get dispatched to nodes that have the infiniband highspeed interconnect, as this will result a much better performance.

Job array

LSF example:

bsub -J "myarray[1-4]" 'echo "Hello, I am task $LSB_JOBINDEX of $LSB_JOBINDEX_END"'

Slurm example:

sbatch --array=1-4 --wrap='echo "Hello, I am task $SLURM_ARRAY_TASK_ID of $SLURM_ARRAY_TASK_COUNT"'

GPU job

For multi-node jobs you need to use the --gpus-per-node option instead.

GPU job requiring a specific GPU model

• GPU model strings for LSF

• For Slurm, currently the specifiers gtx_1080 and rtx_3090 are supported until we add more GPU types.

GPU job requiring a given amount of GPU memory

The default unit for gpumem is bytes. You are therefore advised to specify units, for example 20g or 11000m.

Submit a job using a specific share

In Slurm, one can define a default share using the command: "echo account=es_example >> $HOME/.slurm/defaults"

Submit a job on a specific CPU model

Job chains

Job dependencies

In Slurm, sbatch --parsable returns the JOBID of the job

Job control

Job status

Resource usage

Use --format JobID,User,State,AllocCPUS,Elapsed,NNodes,NTasks,TotalCPU,REQMEM,MaxRSS,ExitCode instead of -l for a customizable, more readable output.

Killing a job

Environment variables