Using the batch system
On our HPC cluster, we use the IBM LSF (Load Sharing Facility) batch system. A basic knowledge of LSF is required if you would like to work on the HPC clusters. The present article will show you how to use LSF to execute simple batch jobs and give you an overview of some advanced features that can dramatically increase your productivity on a cluster.
Using a batch system has numerous advantages:
- single system image — all computing resources in the cluster can be accessed from a single point
- load balancing — the workload is automatically distributed across all available processors
- exclusive use — many computations can be executed at the same time without affecting each other
- prioritization — computing resources can be dedicated to specific applications or people
- fair share — a fair allocation of those resources among all users is guaranteed
In fact, our HPC clusters contains so many processors (30,000) and are used by so many people (more than 2,000) that it would be impossible to use it efficiently without a batch system.
All computations on our HPC cluster must be submitted to the batch system. Please do not run any job interactively on the login nodes, except for testing or debugging purposes.
Basic job submission
Simple commands and programs
Submitting a job to the batch system is as easy as:
bsub command [arguments] example: bsub gzip big_file.dat bsub /path/to/program [arguments] example: bsub ./hello_world
Two or more commands can be combined together by enclosing them in quotes:
bsub "command1; command2" example: bsub "configure; make; make install"
Quotes are also necessary if you want to use I/O redirection (">", "<"), pipes ("|") or conditional operators ("&&", "||"):
bsub "command < data.in > data.out" bsub "command1 | command2"
More complex commands may be placed in a shell script, which should then be submitted like this:
bsub < script
In principle, it is also possible to submit a script as if it were a program:
bsub /path/to/script ← BAD IDEA!
however this syntax is strongly discouraged on Brutus because it does not allow the batch system to "see" what your script is doing, which may lead to errors in the submission and/or execution of your job.
By default your job's output (or standard output, to be precise) is written into a file named lsf.oJobID in the directory where you executed bsub, where JobID is the number assigned to your job by LSF. You can select a different output file using the option:
bsub -o output_file command [argument]
The option -o output_file tells LSF to append your job's output to output_file. If you want to overwrite this file, use:
bsub -oo output_file ...
Note that this option, like all bsub options, must be placed before the command that you want to execute in your job. A common mistake is to place bsub options in the wrong place, like.
bsub command -o output_file ← WRONG!
Batch interactive job
If you just want to run a quick test, you can submit it as a batch interactive job. In this case the job's output is not written into a file, but directly to your terminal, as if it were executed interactively:
bsub -I command [arguments] example: bsub -I "env | sort"