Using the batch system
- 1 Introduction
- 2 Basic job submission
- 3 Resource requirements
- 4 Parallel job submission
- 5 Job control/monitoring
- 6 Troubleshooting
On our HPC cluster, we use the IBM LSF (Load Sharing Facility) batch system. A basic knowledge of LSF is required if you would like to work on the HPC clusters. The present article will show you how to use LSF to execute simple batch jobs and give you an overview of some advanced features that can dramatically increase your productivity on a cluster.
Using a batch system has numerous advantages:
- single system image — all computing resources in the cluster can be accessed from a single point
- load balancing — the workload is automatically distributed across all available processors
- exclusive use — many computations can be executed at the same time without affecting each other
- prioritization — computing resources can be dedicated to specific applications or people
- fair share — a fair allocation of those resources among all users is guaranteed
In fact, our HPC clusters contains so many processors (30,000) and are used by so many people (more than 2,000) that it would be impossible to use it efficiently without a batch system.
All computations on our HPC cluster must be submitted to the batch system. Please do not run any job interactively on the login nodes, except for testing or debugging purposes.
Basic job submission
Simple commands and programs
Submitting a job to the batch system is as easy as:
bsub command [arguments] example: bsub gzip big_file.dat bsub /path/to/program [arguments] example: bsub ./hello_world
Two or more commands can be combined together by enclosing them in quotes:
bsub "command1; command2" example: bsub "configure; make; make install"
Quotes are also necessary if you want to use I/O redirection (">", "<"), pipes ("|") or conditional operators ("&&", "||"):
bsub "command < data.in > data.out" bsub "command1 | command2"
More complex commands may be placed in a shell script, which should then be submitted like this:
bsub < script
In principle, it is also possible to submit a script as if it were a program:
bsub /path/to/script ← BAD IDEA!
however this syntax is strongly discouraged on Brutus because it does not allow the batch system to "see" what your script is doing, which may lead to errors in the submission and/or execution of your job.
By default your job's output (or standard output, to be precise) is written into a file named lsf.oJobID in the directory where you executed bsub, where JobID is the number assigned to your job by LSF. You can select a different output file using the option:
bsub -o output_file command [argument]
The option -o output_file tells LSF to append your job's output to output_file. If you want to overwrite this file, use:
bsub -oo output_file ...
Note that this option, like all bsub options, must be placed before the command that you want to execute in your job. A common mistake is to place bsub options in the wrong place, like.
bsub command -o output_file ← WRONG!
Batch interactive job
If you just want to run a quick test, you can submit it as a batch interactive job. In this case the job's output is not written into a file, but directly to your terminal, as if it were executed interactively:
bsub -I command [arguments] example: bsub -I "env | sort"
By default, a batch job can use only one processor for up to 4 hour. (The job is killed when it reaches its run-time limit.) If your job needs more resources — time, processors, memory or scratch space —, you must request them when you submit it.
The time limits on our clusters are always based on wall-clock (or elapsed) time. You can specify the amount of time needed by your job using the option:
bsub -W minutes ... example: bsub -W 90 ... bsub -W HH:MM ... example: bsub -W 1:30 ...
Since our clusters contains processors with different speeds two similar jobs will not necessarily take the same time to complete. It is therefore safer to request more time than strictly necessary... but not too much, for shorter jobs have generally a higher priority than longer ones.
The maximum run-time for jobs that can run on most compute nodes in the cluster is 240 hours, with one exception. Jobs that run on virtual compute nodes can run for up to 40 days, but there are restrictions in terms of how many cores such jobs can use.
Number of processor cores
If your job requires multiple processors (or threads), you must request them using the option:
bsub -n number_of_procs ...
Note that merely requesting multiple processors does not mean that your application will use them.
By default the batch system allocates 1024 MB (1 GB) of memory per processor core. A single-core job will thus get 1 GB of memory; a 4-core job will get 4 GB; and a 16-core job, 16 GB. If your computation requires more memory, you must request it when you submit your job:
bsub -R "rusage[mem=XXX]" ... example: bsub -R "rusage[mem=2048]"
where XXX is the amount of memory needed by your job, in MB per processor.
LSF automatically creates a local scratch directory when your job starts and deletes it when the job ends. This directory has a unique name, which is passed to your job via the variable $TMPDIR.
Unlike memory, the batch system does not reserve any disk space for this scratch directory by default. If your job is expected to write large amounts of temporary data (say, more than 250 MB) into $TMPDIR — or anywhere in the local /scratch file system — you must request enough scratch space when you submit it:
bsub -R "rusage[scratch=YYY]" ... example: bsub -R "rusage[scratch=10000]" ...
where YYY is the amount of scratch space needed by your job, in MB per processor.
Note that /tmp is reserved for the operating system. Do not write temporary data there! You should either use the directory created by LSF ($TMPDIR) or create your own temporary directory in the local /scratch file system; in the latter case, do not forget to delete this directory at the end of your job.
It is possible to combine memory and scratch requirements:
bsub -R "rusage[mem=XXX]" -R "rusage[scratch=YYY]" ...
is equivalent to:
bsub -R "rusage[mem=XXX,scratch=YYY]" ...
For users that are very experienced with using a batch system, we provide a small helper tool, which simplifies to setup the command for requesting resources from the batch system in order to submit a job.