Job submission

From ScientificComputing
Jump to: navigation, search

< Job management with LSF


Submit a parallel job >

The three ways to access computing resources are

  1. Submit a job through command line
  2. Submit a job through a job bash script using #BSUB pragmas
  3. Get an interactive session on a compute node

Basic job submission

A basic BSUB job submission command consists of three parts:

bsub LSF options Job


bsub   is the LSF submit command.
LSF options   are for requesting resources and defining job-related options.
Job   is a computing job to be submitted.

Here is an example:

bsub -n 1 -W 4:00 -R "rusage[mem=4096]" "python"

When the job is submitted, LSF shows job's information:

$ bsub -n 1 -W 4:00 -R "rusage[mem=4096]" "python"
Generic job.
Job <8146539> is submitted to queue <normal.4h>

The output includes

  1. Job type, e.g., Generic Job, MPI Job or Abaqus Job
  2. Unique JobID, e.g., 8146539
  3. The queue, e.g., normal.4h, normal.24h, or normal.120h

Note: The JobID is important for monitoring or reporting issues. It is also integrated in the name of the LSF output file. Please don't delete the LSF output file unless you are sure that the job was running fine.


A job can be one of the following:

Job Command Examples of job submission command
a single Linux command cmd
a program with its path /path/to/myprogram bsub ./bin/hello
a command or program with its arguments cmd arg1 arg2 bsub echo hello
multiple commands "cmd1 ; cmd2" bsub "date; pwd; ls -l"
piped command "cmd1 | cmd2"
a command with I/O redirection, quote "cmd<in >out" bsub "du -sk /scratch > du.out"
a here document, passed via "<<" << EOF ... EOF
a shell script, passed via "<" < script bsub <

LSF options

Requesting resources

Resources Format Default values
Maximum run time -W HH:MM 04:00 (4h), max. 360h
Number of processors -n nprocs 1 processor
Memory -R "rusage[mem=2048]" 1024 MB per core
Local scratch space -R "rusage[scratch=10000]" 0 MB

It is possible to combine memory and scratch requirements:

-R "rusage[mem=2048, scratch=10000]"

Note: Unlike memory, the batch system does not reserve any disk space for this scratch directory by default. If your job is expected to write large amounts of temporary data (say, more than 250 MB) into $TMPDIR — or anywhere in the local /scratch file system — you must request enough scratch space

Other LSF options

-o outfile append job’s standard output to outfile
-e errfile append job’s error messages to errfile
-R "rusage[...]" advanced resource requirement (memory,...)
-J jobname assign a jobname to the job
-w "depcond" wait until dependency condition is satisfied
-Is submit an interactive job with pseudo-terminal
-B /-N send an email when the job begins/ends
-u user@domain use this address instead of

LSF submission line advisor can assist your to find LSF options you need.

Job script and #BSUB pragmas

Create a job script called job_script.bsub

#BSUB -n 24                     # 24 cores
#BSUB -W 8:00                   # 8-hour run-time
#BSUB -R "rusage[mem=4000]"     # 4000 MB per core
#BSUB -J analysis1
#BSUB -o analysis1.out
#BSUB -e analysis1.err

source /cluster/apps/local/  # Switch to the new software stack
module load gcc/6.3.0 openmpi/4.0.2     # Load modules
cd /path/to/execution/folder            # Change directory
mpirun myprogram arg1                   # Execute the program

Submit a job

$ bsub < job_script.bsub

The options specified on the command line take precedence over the options in the job script.

$ bsub -n 36 < job_script.bsub

Interactive session on a compute node

To run a quick test or a benchmark, you can request an interactive session on a compute node by using the BSUB option

-I, -Ip or -Is

For example:

[jarunanp@eu-login-38 ~]$ bsub -n 4 -W 01:00 -Is bash
Generic job.
Job <161197292> is submitted to queue <normal.4h>.
<<Waiting for dispatch ...>>
<<Starting on eu-ms-001-15>>
[jarunanp@eu-ms-001-15 ~]$

Further reading


< Job management with LSF


Submit a parallel job >