Difference between revisions of "Using the batch system"

From ScientificComputing
Jump to: navigation, search
Line 16: Line 16:
  
 
<noinclude>==Basic job submission==</noinclude><includeonly>===Basic job submission===</includeonly>
 
<noinclude>==Basic job submission==</noinclude><includeonly>===Basic job submission===</includeonly>
 +
 +
<noinclude>===Simple commands and programs===</noinclude><includeonly>====Simple commands and programs====</includeonly>
 +
Submitting a job to the batch system is as easy as:
 +
 +
bsub ''command [arguments]''            example:  bsub gzip big_file.dat
 +
 +
bsub /path/to/''program [arguments]''    example:  bsub ./hello_world
 +
 +
Two or more commands can be combined together by enclosing them in quotes:
 +
 +
bsub "''command1''; ''command2''"            example:  bsub "configure; make; make install"
 +
 +
Quotes are also necessary if you want to use I/O redirection ("<tt>></tt>", "<tt><</tt>"), pipes ("<tt>|</tt>") or conditional operators ("<tt>&&</tt>", "<tt>||</tt>"):
 +
 +
bsub "''command'' < ''data.in'' > ''data.out''"
 +
 +
bsub "''command1'' | ''command2''"
 +
 +
<noinclude>===Shell scripts===</noinclude><includeonly>====Shell scripts====</includeonly>
 +
More complex commands may be placed in a shell script, which should then be submitted like this:
 +
 +
bsub < ''script''
 +
 +
In principle, it is also possible to submit a script as if it were a program:
 +
 +
bsub ''/path/to/script''                &larr;  '''BAD IDEA!'''
 +
 +
however this syntax is '''strongly discouraged''' on Brutus because it does not allow the batch system to "see" what your script is doing, which may lead to errors in the submission and/or execution of your job.
 +
 +
<noinclude>===Output file===</noinclude><includeonly>====Output file====</includeonly>
 +
By default your job's output (or ''standard output'', to be precise) is written into a file named <tt>lsf.o''JobID''</tt>
 +
in the directory where you executed <tt>bsub</tt>, where ''JobID'' is the number
 +
assigned to your job by LSF. You can select a different output file using the option:
 +
 +
bsub -o ''output_file'' ''command [argument]''
 +
 +
The option <tt>-o ''output_file''</tt> tells LSF to '''append''' your job's output to ''output_file''. If you want to '''overwrite''' this file, use:
 +
 +
bsub -oo ''output_file'' ...
 +
 +
Note that this option, like all <tt>bsub</tt> options, must be placed before the ''command'' that you want to execute in your job. A common mistake is to place <tt>bsub</tt> options in the wrong place, like.
 +
 +
bsub ''command'' -o ''output_file''          &larr;  '''WRONG!'''
 +
 +
<noinclude>===Batch interactive job===</noinclude><includeonly>====Batch interactive job====</includeonly>
 +
If you just want to run a quick test, you can submit it as a ''batch interactive'' job. In this case the job's output is not written into a file, but directly to your terminal, as if it were executed interactively:
 +
 +
bsub -I ''command [arguments]''          example:  bsub -I "env | sort"
  
 
<noinclude>==Resource requirements==</noinclude><includeonly>===Resource requirements===</includeonly>
 
<noinclude>==Resource requirements==</noinclude><includeonly>===Resource requirements===</includeonly>

Revision as of 14:25, 23 August 2016

Introduction

On our HPC cluster, we use the IBM LSF (Load Sharing Facility) batch system. A basic knowledge of LSF is required if you would like to work on the HPC clusters. The present article will show you how to use LSF to execute simple batch jobs and give you an overview of some advanced features that can dramatically increase your productivity on a cluster.

Using a batch system has numerous advantages:

  • single system image — all computing resources in the cluster can be accessed from a single point
  • load balancing — the workload is automatically distributed across all available processors
  • exclusive use — many computations can be executed at the same time without affecting each other
  • prioritization — computing resources can be dedicated to specific applications or people
  • fair share — a fair allocation of those resources among all users is guaranteed

In fact, our HPC clusters contains so many processors (30,000) and are used by so many people (more than 2,000) that it would be impossible to use it efficiently without a batch system.

All computations on our HPC cluster must be submitted to the batch system. Please do not run any job interactively on the login nodes, except for testing or debugging purposes.

Basic job submission

Simple commands and programs

Submitting a job to the batch system is as easy as:

bsub command [arguments]             example:  bsub gzip big_file.dat

bsub /path/to/program [arguments]    example:  bsub ./hello_world

Two or more commands can be combined together by enclosing them in quotes:

bsub "command1; command2"            example:  bsub "configure; make; make install"

Quotes are also necessary if you want to use I/O redirection (">", "<"), pipes ("|") or conditional operators ("&&", "||"):

bsub "command < data.in > data.out"

bsub "command1 | command2"

Shell scripts

More complex commands may be placed in a shell script, which should then be submitted like this:

bsub < script

In principle, it is also possible to submit a script as if it were a program:

bsub /path/to/scriptBAD IDEA!

however this syntax is strongly discouraged on Brutus because it does not allow the batch system to "see" what your script is doing, which may lead to errors in the submission and/or execution of your job.

Output file

By default your job's output (or standard output, to be precise) is written into a file named lsf.oJobID in the directory where you executed bsub, where JobID is the number assigned to your job by LSF. You can select a different output file using the option:

bsub -o output_file command [argument]

The option -o output_file tells LSF to append your job's output to output_file. If you want to overwrite this file, use:

bsub -oo output_file ...

Note that this option, like all bsub options, must be placed before the command that you want to execute in your job. A common mistake is to place bsub options in the wrong place, like.

bsub command -o output_fileWRONG!

Batch interactive job

If you just want to run a quick test, you can submit it as a batch interactive job. In this case the job's output is not written into a file, but directly to your terminal, as if it were executed interactively:

bsub -I command [arguments]          example:  bsub -I "env | sort"

Resource requirements

Parallel job submission

Job control/monitoring

Troubleshooting