View source for Using the batch system

<!-- Starts with command summary in order to force the search to fall here for "kill job" -->
<noinclude>==Command Summary==</noinclude><includeonly>===Command Summary===</includeonly>
Please find below a table with commands for job submission, monitoring and control

{| class="wikitable"
! Command !! Description
|-
| <tt>sbatch</tt>
| Submit scripts to Slurm
|-
| <tt>scancel</tt>
| Kill a job
|-
| <tt>srun</tt>
| Run a parallel job within Slurm (e.g. create a job or do it within the current one)
|-
| <tt>squeue</tt>
| View job and job step information for jobs managed by Slurm
|-
| <tt>scontrol</tt>
| Display information about the resource usage of a job
|-
| <tt>sstat</tt>
| Display the status information of a running job/step
|-
| <tt>sacct</tt>
| Displays accounting data for all jobs and job steps in the Slurm job accounting log or Slurm database
|-
| <tt>myjobs</tt>
| Job information in human readable format
|-
|}



<noinclude>==Introduction==</noinclude><includeonly>===Introduction===</includeonly>

On our HPC cluster, we use the '''Slurm''' (Simple Linux Utility for Resource Management) batch system. A basic knowledge of Slurm is required if you would like to work on the HPC clusters of ETH. The present article will show you how to use Slurm to execute simple ''batch jobs'' and give you an overview of some advanced features that can dramatically increase your productivity on a cluster.

Using a batch system has numerous advantages:

* ''single system image'' &mdash; all computing resources in the cluster can be accessed from a single point
* ''load balancing'' &mdash; the workload is automatically distributed across all available processor cores
* ''exclusive use'' &mdash; many computations can be executed at the same time without affecting each other
* ''prioritization'' &mdash; computing resources can be dedicated to specific applications or people
* ''fair share'' &mdash; a fair allocation of those resources among all users is guaranteed

In fact, our HPC clusters contains so many cores (130,000) and are used by so many people (more than 3,200) that it would be '''impossible''' to use it efficiently without a batch system.

All computations on our HPC cluster '''must''' be submitted to the batch system. Please do not run any job interactively on the login nodes, except for testing or debugging purposes.

If you are a member of multiple shareholder groups, then please have a look at our wiki page about [[Multiple_shareholder_groups|working in multiple shareholder groups]]

<noinclude>==Basic job submission==</noinclude><includeonly>===Basic job submission===</includeonly>

We provide a '''helper tool''' to facilitate setting up submission commands and/or jobscript for Slurm and LSF

[https://scicomp.ethz.ch/public/lsla/index2.html '''Slurm/LSF Submission Line Advisor''']

You can specify the resource required by your job and the command and the script will output the corresponding Slurm/LSF submission command or jobscript, depending on your choice.

Slurm provides two different ways of submitting jobs. While we first show the solution with <tt>--wrap</tt>, we strongly recommend to use scripts as indicated in the section [[Using_the_batch_system#Job_scripts|Job scripts]].
The scripts require a bit more work to run a job but comes with some majors advantages:
* Better reproducibility
* More easy and faster handover (which includes the cluster support when you need our help)
* Can load the modules directly within the script

<noinclude>===Simple commands and programs===</noinclude><includeonly>====Simple commands and programs====</includeonly>
Submitting a job to the batch system is as easy as:

 sbatch --wrap="''command [arguments]''"
 sbatch --wrap="/path/to/''program [arguments]''"

Examples:

 [sfux@eu-login-03 ~]$ '''sbatch --wrap="gzip big_file.dat"'''
 Submitted batch job 1010113

 [sfux@eu-login-03 ~]$ '''sbatch --wrap="./hello_world"'''
 Submitted batch job 1010171

Two or more commands can be combined together by enclosing them in quotes:

 sbatch --wrap="''command1''; ''command2''"

Example:

 [sfux@eu-login-03 ~]$ '''sbatch --wrap "configure; make; make install"'''
 Submitted batch job 1010213.

Quotes are also necessary if you want to use I/O redirection ("<tt>></tt>", "<tt><</tt>"), pipes ("<tt>|</tt>") or conditional operators ("<tt>&&</tt>", "<tt>||</tt>"):

 sbatch --wrap="''command'' < ''data.in'' > ''data.out''"
 sbatch --wrap="''command1'' | ''command2''"

Examples:

 [sfux@eu-login-03 ~]$ '''sbatch --wrap="tr ',' '\n' < comma_separated_list > linebreak_separated_list"'''
 Submitted batch job 1010258

 [sfux@eu-login-03 ~]$ '''sbatch --wrap="cat unsorted_list_with_redundant_entries | sort | uniq > sorted_list"'''
 Submitted batch job 1010272

<noinclude>===Shell scripts===</noinclude><includeonly>====Shell scripts====</includeonly>
More complex commands may be placed in a shell script, which should then be submitted like this:

 sbatch < ''script''
 sbatch ''script''

Example:

 [sfux@eu-login-03 ~]$ '''sbatch < hello.sh'''
 Submitted batch job 1010279.

<noinclude>===Output file===</noinclude><includeonly>====Output file====</includeonly>
By default your job's output and error messages (or ''stdout'' and ''stderr'', to be precise) are combined and written into a file named <tt>slurm-''JobID''.out</tt>
in the directory where you executed <tt>sbatch</tt>, where ''JobID'' is the number assigned to your job by Slurm. You can select a different output file using the option:

 sbatch --output=''output_file'' --open-mode=append --wrap="''command [argument]''" 

The option <tt>--output ''output_file''</tt> in combination with <tt>--open-mode=append</tt> tells Slurm to '''append''' your job's output to ''output_file''. If you want to '''overwrite''' this file, use:

 sbatch --output ''output_file'' --open-mode=truncate --wrap="''command [argument]''"

Note that this option, like all <tt>sbatch</tt> options, must be placed before the ''command'' that you want to execute in your job. A common mistake is to place <tt>sbatch</tt> options in the wrong place, like.

 sbatch --wrap=''command'' -o ''output_file''          &larr;  '''WRONG!'''

<noinclude>===Error file===</noinclude><includeonly>====Error file====</includeonly>
It is also possible to store <tt>stderr</tt> of a job in a separate file (and again, you can choose with the --open-mode parameter if you would like to append or overwrite)

 sbatch --error=''error_file'' --open-mode=append --wrap "''command [argument]''"

<noinclude>===Queue / Queues===</noinclude><includeonly>====Queue / Queues====</includeonly>
Slurm uses different queues to manage the scheduling of the jobs. As a user, you don't need to specify which queue to use as it is automatically picked by slurm when you submit the job.

<noinclude>==Resource requirements==</noinclude><includeonly>===Resource requirements===</includeonly>
By default, a batch job can use only '''one core''' for up to '''1 hour'''. (The job is killed when it reaches its run-time limit.) If your job needs more ''resources'' &mdash; time, cores, memory or scratch space &mdash;, you '''must request''' them when you submit it.

<noinclude>===Wall-clock time===</noinclude><includeonly>====Wall-clock time====</includeonly>
The time limits on our clusters are always based on ''wall-clock'' (or ''elapsed'') time. You can specify the amount of time needed by your job with several formats using the option:

 sbatch --time=''minutes'' ...                        example:  sbatch --time=10 ...
 sbatch --time=''minutes:seconds'' ...                example:  sbatch --time=10:50 ...
 sbatch --time=''hours:minutes:seconds'' ...          example:  sbatch --time=5:10:50 ...
 sbatch --time=''days-hours'' ...                     example:  sbatch --time=1-5 ...
 sbatch --time=''days-hours:minutes'' ...             example:  sbatch --time=1-5:10 ...
 sbatch --time=''days-hours:minutes:seconds'' ...     example:  sbatch --time=1-5:10:50 ...

Examples:

 [sfux@eu-login-03 ~]$ '''sbatch --time=20 --wrap="./Riemann_zeta -arg 26"'''
 Submitted batch job 1010305

 [sfux@eu-login-03 ~]$ '''sbatch --time=20:00 --wrap="./solve_Koenigsberg_bridge_problem"'''
 Submitted batch job 1010312.

Since our clusters contains processor cores with different speeds two similar jobs will not necessarily take the same time to complete. It is therefore safer to request more time than strictly necessary... but not too much, for shorter jobs have generally a higher priority than longer ones.

The maximum run-time for jobs that can run on most compute nodes in the cluster is 360 hours. We remain the right to stop jobs with a run time of more than 5 days in case of an emergency maintenance.

<noinclude>===Number of processor cores===</noinclude><includeonly>====Number of processor cores====</includeonly>
If your job requires multiple cores (or threads), you must request them using the option:

 sbatch --ntasks=''number_of_cores'' --wrap="..."

or

 sbatch --ntasks=1 --cpus-per-task=''number_of_cores'' --wrap="..."

'''Please make sure to check the paragraph about [[Using_the_batch_system#Parallel_job_submission|parallel job submission]] before requesting multiple cores.'''

Note that merely ''requesting'' multiple cores does not mean that your application will ''use'' them. 

<noinclude>===Memory===</noinclude><includeonly>====Memory====</includeonly>
By default the batch system allocates 1024 MB (1 GB) of memory per processor core. A single-core job will thus get 1 GB of memory; a 4-core job will get 4 GB; and a 16-core job, 16 GB. If your computation requires more memory, you '''must''' request it when you submit your job:

 sbatch --mem-per-cpu=''XXX'' ...

where ''XXX'' is an integer. The default unit is MB, but you can also specify the value in GB when adding the suffix "G" after the integer value.

Example:

 [sfux@eu-login-03 ~]$ '''sbatch --mem-per-cpu=2G --wrap="./evaluate_gamma -precision 10e-30"'''
 Submitted batch job 1010322

'''Note''': Please note that users cannot request the full memory of a node, as some of the memory is reserved for the operating system of the compute nodes that runs in memory. Therefore if a user for instance requests 256 GiB of memory, then job will not be dispatched to a node with 256 GiB of memory, but on a node with 512 GiB memory or more. As a general rule, jobs that request ~3% less memory than a node has can run on that node type. For instance, on a node with 256 GiB of memory, you can request up to 256*0.97 GiB = 248.32 GiB.

<noinclude>===Scratch space===</noinclude><includeonly>====Scratch space====</includeonly>
Slurm automatically creates a local ''scratch'' directory when your job starts and deletes it when the job ends. This directory has a unique name, which is passed to your job via the variable <tt>$TMPDIR</tt>.

Unlike memory, the batch system '''does not''' reserve any disk space for this scratch directory by default. If your job is expected to write large amounts of temporary data (say, more than 250 MB) into <tt>$TMPDIR</tt> &mdash; or anywhere in the local <tt>/scratch</tt> file system &mdash; you '''must''' request enough scratch space when you submit it:

 sbatch --tmp=''YYY'' ...

where ''YYY''' is the amount of scratch space needed by your job, in MB '''per host''' (there is no setting in Slurm to request it per core). You can also specify the amount in GB by adding the suffix "G" after '''YYY'''.

Example:

 [sfux@eu-login-03 ~]$ '''sbatch --tmp=5000 --wrap="./generating_Euler_numbers -num 5000000"'''
 Submitted batch job 1010713

Note that <tt>/tmp</tt> is '''reserved for the operating system'''. Do not write temporary data there! You should either use the directory created by Slurm (<tt>$TMPDIR</tt>) or create your own temporary directory in the local <tt>/scratch</tt> file system; in the latter case, do not forget to '''delete''' this directory at the end of your job.

<noinclude>===GPU===</noinclude><includeonly>====GPU====</includeonly>
There are GPU nodes in the Euler cluster. The GPU nodes are reserved exclusively to the shareholder groups that invested into them. Guest users and shareholder that purchase CPU nodes but no GPU nodes cannot use the GPU nodes.
 
All GPUs in Slurm are configured in non-exclusive process mode, such that you can run multiple processes/threads on a single GPU. Please find below the available GPU node types.

'''Euler'''
{{GPUTable}}

You can request one or more GPUs with the command

 sbatch --gpus=''number of GPUs'' ...

To run multi-node GPU jobs, you need to use the option <tt>--gpus-per-node</tt>:

 sbatch --gpus-per-node=2 ...

For advanced settings, please have a look at our [[Getting_started_with_GPUs|getting started with GPUs]] page.

<noinclude>==Interactive jobs==</noinclude><includeonly>===Interactive jobs===</includeonly>
If you just want to run a quick test, you can submit it as a ''batch interactive'' job. In this case the job's output is not written into a file, but directly to your terminal, as if it were executed interactively:

 srun --pty ''bash -l''

Please note that the bash option <tt>-l</tt> is required to start a login shell.

Example:

 [sfux@eu-login-35 ~]$ srun --pty bash -l
 srun: job 2040660 queued and waiting for resources
 srun: job 2040660 has been allocated resources
 [sfux@eu-a2p-515 ~]$

For interactive jobs with X11 forwarding enabled, you need to make sure that you login to the cluster with X11 forwarding enabled and then you can run 

 srun [Slurm options] --x11 --pty bash -l

<noinclude>==Parallel job submission==</noinclude><includeonly>===Parallel job submission===</includeonly>
Before submitting parallel jobs, please make sure that your application can run in parallel at all in order to not waste resources by requesting multiple cores for a serial application. Further more, please do a short [[Parallel_efficiency|scaling analysis]] to see how well your code scales in parallel before requesting dozens or hundreds of cores.

<noinclude>===OpenMP===</noinclude><includeonly>====OpenMP====</includeonly>
If your application is parallelized using [[wikipedia:OpenMP|OpenMP]] or linked against a library using OpenMP (Intel MKL, OpenBLAS, etc.), the number of processor cores (or ''threads'') that it can use is controlled by the environment variable <tt>OMP_NUM_THREADS</tt>. This variable must be set '''before''' you submit your job:

 export OMP_NUM_THREADS=''number_of_cores''
 sbatch --ntasks=1 --cpus-per-task=''number_of_cores'' --wrap="..."

NOTE: if <tt>OMP_NUM_THREADS</tt> is not set, your application will either use one core only, or will attempt to use '''all''' cores that it can find. As you are restricted to your jobs resources, all threads will be bound to the cores allocated to your job. Starting more than 1 thread per core will slow down your application as the threads will be fighting to get time on the CPU.

<noinclude>===MPI===</noinclude><includeonly>====MPI====</includeonly>
Three kinds of MPI libraries are available on our cluster: Open MPI (recommended), Intel MPI and MVAPICH2. Before you can submit and execute an MPI job, you '''must''' load the corresponding modules (compiler + MPI, in that order):

 module load ''compiler''
 module load ''mpi_library''

The command used to launch an MPI application is <tt>mpirun</tt>.

Let's assume for example that <tt>hello_world</tt> was compiled with GCC 6.3.0 and linked with Open MPI 4.1.4. The command to execute this job on 4 cores is:

 module load gcc/6.3.0
 module load open_mpi/4.1.4
 sbatch -n 4 --wrap="mpirun ./hello_world"

Note that <tt>mpirun</tt> automatically uses all cores allocated to the job by Slurm. It is therefore not necessary to indicate this number again to the <tt>mpirun</tt> command itself:

 sbatch --ntasks=4 --wrap="mpirun -np 4 ./hello_world"      &larr;  "-np 4" not needed!

<noinclude>===Pthreads and other threaded applications===</noinclude><includeonly>====Pthreads and other threaded applications====</includeonly>
Their behavior is similar to [[#OpenMP|OpenMP]] applications. It is important to limit the number of threads that the application spawns.  There is no standard way to do this, so '''be sure to check the application's documentation on how to do this'''.  Usually a program supports at least one of four ways to limit itself to ''N'' threads:
* it understands the <tt>OMP_NUM_THREADS=N</tt> environment variable,
* it has its own environment variable, such as <tt>GMX_NUM_THREADS=N</tt> for Gromacs,
* it has a command-line option, such as <tt>-nt N</tt> (for Gromacs), or
* it has an input-file option, such as <tt>num_threads N</tt>.
If you are unsure about the program's behavior, please [[Contact|contact us]] and we will analyze it.

<noinclude>===Hybrid jobs===</noinclude><includeonly>====Hybrid jobs====</includeonly>
It is possible to run [[Hybrid_jobs|hybrid jobs that mix MPI and OpenMP]] on our HPC clusters, but this requires a more advanced knowledge of slurm and the hardware.

<noinclude>==Job scripts==</noinclude><includeonly>===Job scripts===</includeonly>
You can also use a job script to specify all sbatch options using #SBATCH pragmas. We strongly recommend to load the modules within the submission script in order improve the reproducibility.

 #!/bin/bash
 
 #SBATCH -n 4
 #SBATCH --time=8:00
 #SBATCH --mem-per-cpu=2000
 #SBATCH --tmp=4000                        # per node!!
 #SBATCH --job-name=analysis1
 #SBATCH --output=analysis1.out
 #SBATCH --error=analysis1.err
 
 module load xyz/123
 command1
 command2

The script can the be submitted as

 sbatch < script

or

 sbatch script

<noinclude>==Job monitoring==</noinclude><includeonly>===Job monitoring===</includeonly>
This section is still work in progress. 

<noinclude>===squeue===</noinclude><includeonly>====squeue====</includeonly>
The <tt>squeue</tt> command allows you to get information about pending, running and shortly finished jobs.

 [sfux@eu-login-41 ~]$ '''squeue'''
              JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
            1433323 normal.4h     wrap     sfux  PD      0:04      1 eu-g1-026-2
            1433322 normal.4h     wrap     sfux  R       0:11      1 eu-a2p-483

You can also check only for running jobs (R) or for pending jobs (PD):

 [sfux@eu-login-41 ~]$ '''squeue -t RUNNING'''
              JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
            1433322 normal.4h     wrap     sfux  R       0:28      1 eu-a2p-483
 [sfux@eu-login-41 ~]$ '''squeue -t PENDING'''
              JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
            1433323 normal.4h     wrap     sfux  PD      0:21      1 eu-g1-026-2
 [sfux@eu-login-41 ~]$ 

An overview on all <tt>squeue</tt> options is available in the <tt>squeue</tt> documentation:

 https://slurm.schedmd.com/squeue.html

<noinclude>===scontrol===</noinclude><includeonly>====scontrol====</includeonly>
The command <tt>scontrol</tt> if one of multiple that allow you to check the information about a running job:

 [sfux@eu-login-15 ~]$ '''scontrol show jobid -dd 1498523'''
 JobId=1498523 JobName=wrap
    UserId=sfux(40093) GroupId=sfux-group(104222) MCS_label=N/A
    Priority=1769 Nice=0 Account=normal/es_hpc QOS=es_hpc/normal
    JobState=RUNNING Reason=None Dependency=(null)
    Requeue=0 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=0:0
    DerivedExitCode=0:0
    RunTime=00:00:38 TimeLimit=01:00:00 TimeMin=N/A
    SubmitTime=2022-10-27T11:44:30 EligibleTime=2022-10-27T11:44:30
    AccrueTime=2022-10-27T11:44:30
    StartTime=2022-10-27T11:44:31 EndTime=2022-10-27T12:44:31 Deadline=N/A
    SuspendTime=None SecsPreSuspend=0 LastSchedEval=2022-10-27T11:44:31 Scheduler=Main
    Partition=normal.4h AllocNode:Sid=eu-login-15:26645
    ReqNodeList=(null) ExcNodeList=(null)
    NodeList=eu-a2p-528
    BatchHost=eu-a2p-528
    NumNodes=1 NumCPUs=1 NumTasks=1 CPUs/Task=1 ReqB:S:C:T=0:0:*:*
    TRES=cpu=1,mem=1G,node=1,billing=1
    Socks/Node=* NtasksPerN:B:S:C=0:0:*:1 CoreSpec=*
    JOB_GRES=(null)
      Nodes=eu-a2p-528 CPU_IDs=127 Mem=1024 GRES=
    MinCPUsNode=1 MinMemoryCPU=1G MinTmpDiskNode=0
    Features=(null) DelayBoot=00:00:00
    OverSubscribe=OK Contiguous=0 Licenses=(null) Network=(null)
    Command=(null)
    WorkDir=/cluster/home/sfux
    StdErr=/cluster/home/sfux/slurm-1498523.out
    StdIn=/dev/null
    StdOut=/cluster/home/sfux/slurm-1498523.out
    Power=

<noinclude>====Squeue States====</noinclude><includeonly>=====Squeue States=====</includeonly>
{| class="wikitable"
! Job Status !! Description
|-
| <tt>QOSMaxCpuPerUserLimit</tt>
| You are using more CPUs than allowed by your share. You can either cancel a running job or wait until they are finished.
|-
| <tt>QOSMaxMemoryPerUser</tt>
| You are using more RAM than allowed by your share. You can either cancel a running job or wait until they are finished.
|-
| <tt>QOSMaxGRESPerUser</tt>
| You are using more generic resources (e.g. GPUs) than allowed by your share. You can either cancel a running job or wait until they are finished.
|-
| <tt>PartitionDown</tt>
| If a maintenance will start soon, some partitions will not be available. Otherwise, it might be an issue on our side.
|-
| <tt>Priority</tt>
| Your job is scheduled, but some other jobs with a higher priority (e.g. that have been longer in the queue) are scheduled before yours.
|-
| <tt>ReqNodeNotAvail</tt>
| Your job requirements cannot match any available nodes. Either wait until some resources are available or reduce your restriction (e.g. RAM, cores, GPU type, GPU RAM, ...).
|-
| <tt>Nodes required for job are DOWN, DRAINED or reserved for jobs in higher priority partitions</tt>
| Your job requirements cannot match any available nodes. Either wait until some resources are available or reduce your restriction (e.g. RAM, cores, GPU type, GPU RAM, ...).
|-
| <tt>Resources</tt>
| Your job is waiting for resources to be available.
|-
| <tt>InvalidAccount</tt>
| If you don't specify the account when submitting your job, you might have been removed from a share. Please try to logout / login to update the share information. Otherwise, please check that the account name is not invalid with `my_share_info`.
|-
| <tt>PartitionTimeLimit</tt>
| Your job requests more time than available for the partition.
|-
| <tt>JobArrayTaskLimit</tt>
| Too many jobs in an array are already running, waiting will solve the issue.
|-
|}



<noinclude>===sstat===</noinclude><includeonly>====sstat====</includeonly>
You can use the <tt>sstat</tt> command to diplay information about your running jobs, for instance resources like CPU time (MinCPU) and memory usage (MaxRSS):

 [sfux@eu-login-35 ~]$ '''sstat --all --format JobID,NTasks,MaxRSS,MinCPU -j 2039738'''
 JobID          NTasks     MaxRSS     MinCPU
 ------------ -------- ---------- ----------
 2039738.ext+        1          0   00:00:00
 2039738.bat+        1    886660K   00:07:14

An overview on all available fields for the format option is provided in the <tt>sstat</tt> documentation

 https://slurm.schedmd.com/sstat.html

<noinclude>===sacct===</noinclude><includeonly>====sacct====</includeonly>
The sacct command allows users to check information on running or finished jobs.

 [sfux@eu-login-35 ~]$ '''sacct  --format JobID,User,State,AllocCPUS,Elapsed,NNodes,NTasks,ReqMem,ExitCode'''
 JobID             User      State  AllocCPUS    Elapsed   NNodes   NTasks     ReqMem ExitCode
 ------------ --------- ---------- ---------- ---------- -------- -------- ---------- --------
 2039738           sfux    RUNNING          4   00:06:01        1                  8G      0:0
 2039738.bat+              RUNNING          4   00:06:01        1        1                 0:0
 2039738.ext+              RUNNING          4   00:06:01        1        1                 0:0
 [sfux@eu-login-35 ~]$

An overview on all format fields for the <tt>sacct</tt> is available in the documentation

 https://slurm.schedmd.com/sacct.html

Please note that the CPU time (TotalCPU) and memory usage (MaxRSS) are only correctly displayed for finished jobs. If you check this properties for running jobs, then it will just show 0. For checking the CPU time and memory usage of running jobs, please use <tt>sstat</tt>. 

<noinclude>===myjobs===</noinclude><includeonly>====myjobs====</includeonly>
We are working on providing a <tt>bbjobs</tt> like wrapper for monitoring Slurm jobs. The wrapper script is called <tt>myjobs</tt> and accepts a single option <tt>-j</tt> to specify the jobid.

* Please note that the script only correctly works for simple jobs without additional job steps
* Please note that using the cat emoji 🐈 in a user command will cause the myjobs script to fail
* Please note that the CPU and memory efficiency for multi-node jobs displayed by myjobs is not correct (CPU time and memory usage reports the CPU time for only 1 node out of the resource allocation).

The script is still work in progress and we try to improve it continuously.

 [sfux@eu-login-39 ~]$ myjobs -j 2647208
 Job information
  Job ID                          : 2647208
  Status                          : RUNNING
  Running on node                 : eu-a2p-277
  User                            : sfux
  Shareholder group               : es_hpc
  Slurm partition (queue)         : normal.24h
  Command                         : sbatch --ntasks=4 --time=4:30:00 --mem-per-cpu=2g
  Working directory               : /cluster/home/sfux/testrun/adf/2021_test
 Requested resources
  Requested runtime               : 04:30:00
  Requested cores (total)         : 4
  Requested nodes                 : 1
  Requested memory (total)        : 8192 MiB
  Requested scratch (per node)    : #not yet implemented#
 Job history
  Submitted at                    : 2022-11-18T11:10:37
  Started at                      : 2022-11-18T11:10:37
  Queue waiting time              : 0 sec
 Resource usage
  Wall-clock                      : 00:10:34
  Total CPU time                  : 00:41:47
  CPU utilization                 : 98.85%
  Total resident memory           : 1135.15 MiB
  Resident memory utilization     : 13.85%
 [sfux@eu-login-39 ~]$ 

We are still working on implementing some missing features like displaying the requested local scratch and Sys/Kernel time.

If you would like to get the myjobs output for all your jobs in the queue (pending/running), you can omit the jobid as parameter:

 myjobs

for displaying only information about pending jobs, you can use

 myjobs -p

for displaying only information about running jobs, you can use

 myjobs -r

Please note that these commands might not work for job arrays.

<noinclude>===scancel===</noinclude><includeonly>====scancel====</includeonly>
You can use the scancel to cancel jobs.

 [sfux@eu-login-15 ~]$ '''squeue'''
 JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
 1525589 normal.24 sbatch sfux R 0:11 1 eu-a2p-373
 [sfux@eu-login-15 ~]$ '''scancel 1525589'''
 [sfux@eu-login-15 ~]$ '''squeue'''
 JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
 [sfux@eu-login-15 ~]$

<noinclude>===bjob_connect===</noinclude><includeonly>====bjob_connect====</includeonly>
Sometimes it is necessary to monitor the job on the node(s) where it is running.
On Euler, compute nodes can not be accessed directly via ssh. To access a node where a job is running the tool srun should be used.
You can connect to one of your running job with srun.

 srun --interactive --jobid JOBID --pty bash

where you need to replace JOBID with the id of your batch job. For jobs running on multiple nodes, you can use <tt>--nodelist=NODE</tt> to pick one.