GPU job submission with SLURM

From ScientificComputing
Jump to: navigation, search

< Submit a parallel job


Monitor a job >

ⓘ Note

You can only use GPUs if you are a member of a shareholder group that invested into GPU nodes

Cpu gpu system arch.png

Figure: Here is an example of CPU & GPU system architecture. There are several system architectures on the cluster.

To use the GPUs for a job node you need to request the -G or --gpus resource.

For example, to run a serial job with one GPU :

$ sbatch -G 1 --wrap="./my_cuda_program"

How to select GPU memory

If you know that you will need more memory on a GPU than some models provide, i.e., more than 8 GB, then you can request that your job will run only on GPUs that have enough memory. Use the --gres=gpumem: option to do this. For example, if you need 10 GB (=10240  MB) per GPU:

 $ sbatch -G 1 --gres=gpumem:10g --wrap="./my_cuda_program"

This ensures your job will not run on GPUs with less than 10 GB of GPU memory. The default unit for the gpumem option is bytes. You are therefore advised to specify units, for example 20g or 11000m.

How to select a GPU model

In some cases it is desirable or necessary to select the GPU model on which your job runs, for example if you know you code runs much faster on a newer model. However, you should consider that by narrowing down the list of allowable GPUs, your job may need to wait for a longer time.

To select a certain GPU model, add the --gpus=slurm_specifierl:N resource requirement to sbatch, where the SLURM specifier for the GPU models is detailed in the table below, and N is the number of requested GPUs.

$ sbatch -G 1 --gpus=gtx_1080:1 --wrap="./my_cuda_program"


Available GPU node types

GPU Model LSF Specifier (GPU driver > 450.80.02) Slurm specifier GPU memory per GPU CPU cores per node CPU memory per node
NVIDIA GeForce GTX 1080 NVIDIAGeForceGTX1080 gtx_1080 8 GiB 20 256 GiB
NVIDIA GeForce GTX 1080 Ti NVIDIAGeForceGTX1080Ti gtx_1080_ti 11 GiB 20 256 GiB
NVIDIA GeForce RTX 2080 Ti NVIDIAGeForceRTX2080Ti rtx_2080_ti 11 GiB 36 384 GiB
NVIDIA GeForce RTX 2080 Ti NVIDIAGeForceRTX2080Ti rtx_2080_ti 11 GiB 128 512 GiB
NVIDIA GeForce RTX 3090 NVIDIAGeForceRTX3090 rtx_3090 24 GiB 128 512 GiB
NVIDIA Quadro RTX 6000 QuadroRTX6000 quadro_rtx_6000 24 GiB 128 512 GiB
NVIDIA Tesla V100-SXM2 32 GiB TeslaV100_SXM2_32GB v100 32 GiB 48 768 GiB
NVIDIA Tesla V100-SXM2 32 GB TeslaV100_SXM2_32GB v100 32 GiB 40 512 GiB
Nvidia Tesla A100 (40 GiB) NVIDIAA100_PCIE_40GB a100-pcie-40gb 40 GiB 48 768 GiB
Nvidia Tesla A100 (80 GiB) unavailable a100_80gb 80 GiB 48 1024 GiB


Further reading

< Submit a parallel job


Monitor a job >