Difference between revisions of "Job submission"
From ScientificComputing
| (9 intermediate revisions by the same user not shown) | |||
| Line 6: | Line 6: | ||
</td> | </td> | ||
<td style="width: 35%; text-align:center"> | <td style="width: 35%; text-align:center"> | ||
| − | [[ | + | [[Main_Page | Home]] |
</td> | </td> | ||
<td style="width: 35%; text-align:right"> | <td style="width: 35%; text-align:right"> | ||
| Line 16: | Line 16: | ||
The three ways to access computing resources are | The three ways to access computing resources are | ||
| − | # [[Job_submission#Basic_job_submission|Submit a job through command line]] | + | # [[Job_submission#Basic_job_submission|'''Submit a job through command line''']] |
| − | # [[Job_submission#Job_script_and_#BSUB_pragmas|Submit a job through a job bash script using #BSUB pragmas]] | + | # [[Job_submission#Job_script_and_#BSUB_pragmas|'''Submit a job through a job bash script using #BSUB pragmas''']] |
| − | # [[Job_submission#Interactive_session_on_a_compute_node|Get an interactive session on a compute node]] | + | # [[Job_submission#Interactive_session_on_a_compute_node|'''Get an interactive session on a compute node''']] |
== Basic job submission == | == Basic job submission == | ||
| Line 91: | Line 91: | ||
==== Requesting resources ==== | ==== Requesting resources ==== | ||
{| class="wikitable" | style="background:white;" | {| class="wikitable" | style="background:white;" | ||
| − | ! Resources || Format || Default values | + | ! Resources || Format || Default values |
|- | |- | ||
| − | | Maximum run time || -W HH:MM || 04:00 ( | + | | Maximum run time || -W HH:MM || 04:00 (4h), max. 360h |
|- | |- | ||
| Number of processors || -n nprocs || 1 processor | | Number of processors || -n nprocs || 1 processor | ||
| Line 101: | Line 101: | ||
| Local scratch space || -R "rusage[scratch=10000]" || 0 MB | | Local scratch space || -R "rusage[scratch=10000]" || 0 MB | ||
|} | |} | ||
| + | |||
| + | It is possible to combine memory and scratch requirements: | ||
| + | |||
| + | -R "rusage[mem=2048, scratch=10000]" | ||
| + | |||
| + | '''Note''': Unlike memory, the batch system '''does not''' reserve any disk space for this scratch directory by default. If your job is expected to write large amounts of temporary data (say, more than 250 MB) into $TMPDIR — or anywhere in the local /scratch file system — you '''must''' request enough scratch space | ||
==== Other LSF options ==== | ==== Other LSF options ==== | ||
| Line 135: | Line 141: | ||
#BSUB -N | #BSUB -N | ||
| − | module load gcc/6.3.0 openmpi/ | + | source /cluster/apps/local/env2lmod.sh # Switch to the new software stack |
| − | cd /path/to/execution/folder | + | module load gcc/6.3.0 openmpi/4.0.2 # Load modules |
| − | mpirun myprogram arg1 | + | cd /path/to/execution/folder # Change directory |
| + | mpirun myprogram arg1 # Execute the program | ||
Submit a job | Submit a job | ||
$ bsub < job_script.bsub | $ bsub < job_script.bsub | ||
| + | |||
| + | The options specified on the command line take precedence over the options in the job script. | ||
| + | |||
| + | $ bsub -n 36 < job_script.bsub | ||
== Interactive session on a compute node == | == Interactive session on a compute node == | ||
| Line 158: | Line 169: | ||
* [[Job arrays]] | * [[Job arrays]] | ||
* [[Job chaining]] | * [[Job chaining]] | ||
| + | * [[X11 forwarding batch interactive jobs ]] | ||
| + | * [[Multiple shareholder groups]] | ||
* [[LSF_mini_reference|LSF mini reference]] | * [[LSF_mini_reference|LSF mini reference]] | ||
| Line 170: | Line 183: | ||
</td> | </td> | ||
<td style="width: 35%; text-align:center"> | <td style="width: 35%; text-align:center"> | ||
| − | [[ | + | [[Main_Page | Home]] |
</td> | </td> | ||
<td style="width: 35%; text-align:right"> | <td style="width: 35%; text-align:right"> | ||
Latest revision as of 15:06, 8 November 2021
The three ways to access computing resources are
- Submit a job through command line
- Submit a job through a job bash script using #BSUB pragmas
- Get an interactive session on a compute node
Basic job submission
A basic BSUB job submission command consists of three parts:
| bsub | LSF options | Job |
where
| bsub | is the LSF submit command. |
| LSF options | are for requesting resources and defining job-related options. |
| Job | is a computing job to be submitted. |
Here is an example:
| bsub | -n 1 -W 4:00 -R "rusage[mem=4096]" | "python myscript.py" |
When the job is submitted, LSF shows job's information:
$ bsub -n 1 -W 4:00 -R "rusage[mem=4096]" "python myscript.py" Generic job. Job <8146539> is submitted to queue <normal.4h>
The output includes
- Job type, e.g., Generic Job, MPI Job or Abaqus Job
- Unique JobID, e.g., 8146539
- The queue, e.g., normal.4h, normal.24h, or normal.120h
Note: The JobID is important for monitoring or reporting issues. It is also integrated in the name of the LSF output file. Please don't delete the LSF output file unless you are sure that the job was running fine.
| Job |
A job can be one of the following:
| Job | Command | Examples of job submission command |
|---|---|---|
| a single Linux command | cmd | |
| a program with its path | /path/to/myprogram | bsub ./bin/hello |
| a command or program with its arguments | cmd arg1 arg2 | bsub echo hello |
| multiple commands | "cmd1 ; cmd2" | bsub "date; pwd; ls -l" |
| piped command | "cmd1 | cmd2" | |
| a command with I/O redirection, quote | "cmd<in >out" | bsub "du -sk /scratch > du.out" |
| a here document, passed via "<<" | << EOF ... EOF | |
| a shell script, passed via "<" | < script | bsub < hello.sh |
| LSF options |
Requesting resources
| Resources | Format | Default values |
|---|---|---|
| Maximum run time | -W HH:MM | 04:00 (4h), max. 360h |
| Number of processors | -n nprocs | 1 processor |
| Memory | -R "rusage[mem=2048]" | 1024 MB per core |
| Local scratch space | -R "rusage[scratch=10000]" | 0 MB |
It is possible to combine memory and scratch requirements:
-R "rusage[mem=2048, scratch=10000]"
Note: Unlike memory, the batch system does not reserve any disk space for this scratch directory by default. If your job is expected to write large amounts of temporary data (say, more than 250 MB) into $TMPDIR — or anywhere in the local /scratch file system — you must request enough scratch space
Other LSF options
| -o outfile | append job’s standard output to outfile |
| -e errfile | append job’s error messages to errfile |
| -R "rusage[...]" | advanced resource requirement (memory,...) |
| -J jobname | assign a jobname to the job |
| -w "depcond" | wait until dependency condition is satisfied |
| -Is | submit an interactive job with pseudo-terminal |
| -B /-N | send an email when the job begins/ends |
| -u user@domain | use this address instead of username@ethz.ch |
LSF submission line advisor can assist your to find LSF options you need.
Job script and #BSUB pragmas
Create a job script called job_script.bsub
#!/bin/bash #BSUB -n 24 # 24 cores #BSUB -W 8:00 # 8-hour run-time #BSUB -R "rusage[mem=4000]" # 4000 MB per core #BSUB -J analysis1 #BSUB -o analysis1.out #BSUB -e analysis1.err #BSUB -N source /cluster/apps/local/env2lmod.sh # Switch to the new software stack module load gcc/6.3.0 openmpi/4.0.2 # Load modules cd /path/to/execution/folder # Change directory mpirun myprogram arg1 # Execute the program
Submit a job
$ bsub < job_script.bsub
The options specified on the command line take precedence over the options in the job script.
$ bsub -n 36 < job_script.bsub
Interactive session on a compute node
To run a quick test or a benchmark, you can request an interactive session on a compute node by using the BSUB option
-I, -Ip or -Is
For example:
[jarunanp@eu-login-38 ~]$ bsub -n 4 -W 01:00 -Is bash Generic job. Job <161197292> is submitted to queue <normal.4h>. <<Waiting for dispatch ...>> <<Starting on eu-ms-001-15>> [jarunanp@eu-ms-001-15 ~]$
Further reading
- User guide: Using the batch system
- Job arrays
- Job chaining
- X11 forwarding batch interactive jobs
- Multiple shareholder groups
- LSF mini reference
Helper
