View source for GPU job submission with SLURM

__NOTOC__
<table style="width: 100%;">
<tr valign=top>
<td style="width: 30%; text-align:left">
< [[Parallel job submission with SLURM| Submit a parallel job]]
</td>
<td style="width: 35%; text-align:center">
[[Main Page | Home]]
</td>
<td style="width: 35%; text-align:right">
[[Job monitoring with SLURM| Monitor a job]] >
</td>
</tr>
</table>


<div class="button" style=" width: 95%; text-align:left; background: #FFFFCE; border-radius: 5px; padding-left: 15px;">
    <p style="font-weight: bold; size:5">&#9432; Note</p>
    
<font color="#3A3B3C" size="4"> You can only use GPUs if you are a member of a shareholder group that invested into GPU nodes</font>
</div>


<table style="width: 100%;">
<tr valign=top>
<td style="width: 23%; text-align:left">
[[File:Cpu_gpu_system_arch.png|320px]]

<font color="#3A3B3C" size="1"> Figure: Here is an example of CPU & GPU system architecture. There are several system architectures on the cluster.   </font>
</td>
<td style="width: 2%;">
</td>
<td style="width: 75%;">

To use the GPUs for a job node you need to request the '''-G or --gpus''' resource.

For example, to run a serial job with one GPU :
 $ sbatch '''-G 1''' --wrap="./my_cuda_program"

== How to select GPU memory ==

If you know that you will need more memory on a GPU than some models provide, <em>i.e.,</em> more than 8&nbsp;GB, then you can request that your job will run only on GPUs that have enough memory. Use the <tt>--gres=gpumem:</tt> option to do this. For example, if you need 10&nbsp;GB (=10240&nbsp; MB) per&nbsp;GPU:

  $ sbatch -G 1 '''--gres=gpumem:10g''' --wrap="./my_cuda_program"

This ensures your job will not run on GPUs with less than 10&nbsp;GB of GPU memory. 
The default unit for the gpumem option is bytes. You are therefore advised to specify units, for example 20g or 11000m.

== How to select a GPU model ==
In some cases it is desirable or necessary to select the GPU model on which your job runs, for example if you know you code runs much faster on a newer model. However, you should consider that by narrowing down the list of allowable GPUs, your job may need to wait for a longer time.

To select a certain GPU model, add the <tt> --gpus=slurm_specifierl:N </tt> resource requirement to sbatch, where the SLURM specifier for the GPU models is detailed in the table below, and N is the number of requested GPUs.

 $ sbatch -G 1 '''--gpus=gtx_1080:1''' --wrap="./my_cuda_program"

<!-- While your jobs will see all GPUs, LSF will set the [https://devblogs.nvidia.com/parallelforall/cuda-pro-tip-control-gpu-visibility-cuda_visible_devices/ CUDA_VISIBLE_DEVICES] environment variable, which is honored by CUDA -->programs.
</td>
</tr>
</table>



==Available GPU node types==
{{GPUTable}}

== Example ==
* [[Neural network training with TensorFlow on GPU | Deep learning with TensorFlow on GPU]]

== Further reading ==
* [[Getting started with GPUs]]



<table style="width: 100%;">
<tr valign=top>
<td style="width: 30%; text-align:left">
< [[Parallel job submission with SLURM | Submit a parallel job]]
</td>
<td style="width: 35%; text-align:center">
[[Main Page | Home]]
</td>
<td style="width: 35%; text-align:right">
[[Job monitoring with SLURM | Monitor a job]] >
</td>
</tr>
</table>