Difference between revisions of "Using the batch system"

Revision as of 12:41, 30 August 2016

Introduction

On our HPC cluster, we use the IBM LSF (Load Sharing Facility) batch system. A basic knowledge of LSF is required if you would like to work on the HPC clusters. The present article will show you how to use LSF to execute simple batch jobs and give you an overview of some advanced features that can dramatically increase your productivity on a cluster.

Using a batch system has numerous advantages:

• single system image — all computing resources in the cluster can be accessed from a single point
• load balancing — the workload is automatically distributed across all available processors
• exclusive use — many computations can be executed at the same time without affecting each other
• prioritization — computing resources can be dedicated to specific applications or people
• fair share — a fair allocation of those resources among all users is guaranteed

In fact, our HPC clusters contains so many processors (30,000) and are used by so many people (more than 2,000) that it would be impossible to use it efficiently without a batch system.

All computations on our HPC cluster must be submitted to the batch system. Please do not run any job interactively on the login nodes, except for testing or debugging purposes.

Basic job submission

Simple commands and programs

Submitting a job to the batch system is as easy as:

bsub command [arguments]
bsub /path/to/program [arguments] 

Examples:

[leonhard@euler03 ~]$ bsub gzip big_file.dat
Generic job.
Job <8146539> is submitted to queue <normal.4h>.

[leonhard@euler03 ~]$ bsub ./hello_world
Generic job.
Job <8146540> is submitted to queue <normal.4h>.

Two or more commands can be combined together by enclosing them in quotes:

bsub "command1; command2"

Example:

[leonhard@euler03 ~]$ bsub "configure; make; make install"
Generic job.
Job <8146541> is submitted to queue <normal.4h>.

Quotes are also necessary if you want to use I/O redirection (">", "<"), pipes ("|") or conditional operators ("&&", "||"):

bsub "command < data.in > data.out"
bsub "command1 | command2"

Examples:

[leonhard@euler03 ~]$ bsub "tr ',' '\n' < comma_separated_list > linebreak_separated_list"
Generic job.
Job <8146542> is submitted to queue <normal.4h>.

[leonhard@euler03 ~]$ bsub "cat unsorted_list_with_redundant_entries | sort | uniq > sorted_list"
Generic job.
Job <8146543> is submitted to queue <normal.4h>.

Shell scripts

More complex commands may be placed in a shell script, which should then be submitted like this:

bsub < script

Example:

[leonhard@euler03 ~]$ bsub < hello.sh
Generic job.
Job <8146544> is submitted to queue <normal.4h>.

In principle, it is also possible to submit a script as if it were a program:

bsub /path/to/script                 ←  BAD IDEA!

however this syntax is strongly discouraged on our clusters because it does not allow the batch system to "see" what your script is doing, which may lead to errors in the submission and/or execution of your job.

Output file

By default your job's output (or standard output, to be precise) is written into a file named lsf.oJobID in the directory where you executed bsub, where JobID is the number assigned to your job by LSF. You can select a different output file using the option:

bsub -o output_file command [argument]

The option -o output_file tells LSF to append your job's output to output_file. If you want to overwrite this file, use:

bsub -oo output_file ...

Note that this option, like all bsub options, must be placed before the command that you want to execute in your job. A common mistake is to place bsub options in the wrong place, like.

bsub command -o output_file          ←  WRONG!

Batch interactive job

If you just want to run a quick test, you can submit it as a batch interactive job. In this case the job's output is not written into a file, but directly to your terminal, as if it were executed interactively:

bsub -I command [arguments]

Example:

[leonhard@euler03 ~]$ bsub -I "env | sort"
Generic job.
Job <8146545> is submitted to queue <normal.4h>.
<<Waiting for dispatch ...>

Resource requirements

By default, a batch job can use only one processor for up to 4 hour. (The job is killed when it reaches its run-time limit.) If your job needs more resources — time, processors, memory or scratch space —, you must request them when you submit it.

Wall-clock time

The time limits on our clusters are always based on wall-clock (or elapsed) time. You can specify the amount of time needed by your job using the option:

bsub -W minutes ...                  example:  bsub -W 90 ...

bsub -W HH:MM ...                    example:  bsub -W 1:30 ...

Since our clusters contains processors with different speeds two similar jobs will not necessarily take the same time to complete. It is therefore safer to request more time than strictly necessary... but not too much, for shorter jobs have generally a higher priority than longer ones.

The maximum run-time for jobs that can run on most compute nodes in the cluster is 240 hours, with one exception. Jobs that run on virtual compute nodes can run for up to 40 days, but there are restrictions in terms of how many cores such jobs can use.

Number of processor cores

If your job requires multiple processors (or threads), you must request them using the option:

bsub -n number_of_procs ...

Note that merely requesting multiple processors does not mean that your application will use them.

Memory

By default the batch system allocates 1024 MB (1 GB) of memory per processor core. A single-core job will thus get 1 GB of memory; a 4-core job will get 4 GB; and a 16-core job, 16 GB. If your computation requires more memory, you must request it when you submit your job:

bsub -R "rusage[mem=XXX]" ...        example:  bsub -R "rusage[mem=2048]"

where XXX is the amount of memory needed by your job, in MB per processor.

Scratch space

LSF automatically creates a local scratch directory when your job starts and deletes it when the job ends. This directory has a unique name, which is passed to your job via the variable $TMPDIR.

Unlike memory, the batch system does not reserve any disk space for this scratch directory by default. If your job is expected to write large amounts of temporary data (say, more than 250 MB) into $TMPDIR — or anywhere in the local /scratch file system — you must request enough scratch space when you submit it:

bsub -R "rusage[scratch=YYY]" ...    example:  bsub -R "rusage[scratch=10000]" ...

where YYY is the amount of scratch space needed by your job, in MB per processor.

Note that /tmp is reserved for the operating system. Do not write temporary data there! You should either use the directory created by LSF ($TMPDIR) or create your own temporary directory in the local /scratch file system; in the latter case, do not forget to delete this directory at the end of your job.

Multiple requirements

It is possible to combine memory and scratch requirements:

bsub -R "rusage[mem=XXX]" -R "rusage[scratch=YYY]" ...

is equivalent to:

bsub -R "rusage[mem=XXX,scratch=YYY]" ...

LSF submission line advisor

For users that are not yet very experienced with using a batch system, we provide a small helper tool, which simplifies to setup the command for requesting resources from the batch system in order to submit a job.

https://scicomp.ethz.ch/lsf_submission_line_advisor

Parallel job submission

Before submitting parallel jobs, please make sure that your application can run in parallel at all in order to not waste resources by requesting multiple cores for a serial application. Further more, please do a short scaling study to see how well your code scales in parallel before requesting dozens or hundreds of cores. You can calculate the CPU efficiency of your job using a script, that requires the name of the LSF output file and the number of cores as input arguments.

cpu_efficiency.sh:

#!/bin/bash

# cpu_efficiency.sh
# Samuel Fux, 2016 @ ETH Zurich
#
# input variables:
#
# $1 -> file name of the LSF logfile
# $2 -> number of cores used in the job


# get start and end time from lsf.o***** file
START_TIME_STRING=`grep 'Started at' $1`
END_TIME_STRING=`grep 'reported' $1`
# convert dates to UNIX timestamp and calculate difference
WALL_TIME=$(( $(( `date -d "${END_TIME_STRING[*]:20:24}" +%s` ))-$(( `date -d "${START_TIME_STRING[*]:11:24}" +%s` )) ))
# get string that contains CPU time of the job
CPU_TIME_STRING=`grep 'CPU time' $1`
# filter out CPU time in seconds from the string
CPU_TIME_SELECTION=`echo $CPU_TIME_STRING | cut -d ' ' -f 4`
# calculated CPU efficiency
CPU_EFFICIENCY=`echo "scale=2;(100*$CPU_TIME_SELECTION)/($WALL_TIME*$2) " | bc -l` 

echo "Cores          : $2"
echo "Wall time      : $WALL_TIME"
echo "CPU time       : $CPU_TIME_SELECTION"
echo "CPU efficiency : $CPU_EFFICIENCY %" 

The script displays the number of cores, that you have specified, the CPU time used, the wall-clock time and the CPU efficiency.

[leonhard@euler01 ~]$ ls lsf.o7805617 
lsf.o7805617
[leonhard@euler01 ~]$ ./cpu_efficiency.sh lsf.o7805617 4
Cores          : 4
Wall time      : 6272
CPU time       : 24834.58
CPU efficiency : 98.98 %

OpenMP

If your application is parallelized using OpenMP or linked against a library using OpenMP (Intel MKL, OpenBLAS, etc.), the number of processors (or threads) that it can use is controlled by the environment variable OMP_NUM_THREADS. This variable must be set before you submit your job:

export OMP_NUM_THREADS=number_of_processors
bsub -n number_of_processors ...

NOTE: if OMP_NUM_THREADS is not set, your application will either use one processor only, or will attempt to use all processors that it can find, stealing them from other jobs if needed. In other words, your job will either use too few or too many processors.

MPI

Two kinds of MPI libraries are available on our cluster: Open MPI (recommended) and MVAPICH2. Before you can submit and execute an MPI job, you must load the corresponding modules (compiler + MPI, in that order):

module load compiler
module load mpi_library

The command used to launch an MPI application is mpirun.

Let's assume for example that hello_world was compiled with PGI 15.1 and linked with Open MPI 1.6.5. The command to execute this job on 4 processors is:

module load pgi/15.1
module load open_mpi/1.6.5
bsub -n 4 mpirun ./hello_world

Note that mpirun automatically uses all processors allocated to the job by LSF. It is therefore not necessary to indicate this number again to the mpirun command itself:

bsub -n 4 mpirun -np 4 ./hello_world      ←  "-np 4" not needed!

Pthreads and other threaded applications

Their behavior is similar to OpenMP applications. It is important to limit the number of threads that the application spawns. There is no standard way to do this, so be sure to check the application's documentation on how to do this. Usually a program supports at least one of four ways to limit itself to N threads:

• it understands the OMP_NUM_THREADS=N environment variable,
• it has its own environment variable, such as GMX_NUM_THREADS=N for Gromacs,
• it has a command-line option, such as -nt N (for Gromacs), or
• it has an input-file option, such as num_threads N.

If you are unsure about the program's behavior, please Contact us and we will analyze it.

Hybrid jobs

It is possible to run hybrid jobs that mix MPI and OpenMP on our HPC clusters, but we strongly recommend to not submit these kind of jobs,

Job control/monitoring

Troubleshooting