Getting started with clusters
Contents
- 1 Requesting an account
- 2 Accessing the clusters
- 3 Data management
- 4 Setting up the Environment
- 5 Using the batch system
- 6 Applications
Requesting an account
Brutus
Brutus (2007-2016) is no longer in operation.
Euler
Everybody at ETH Zurich can use the Euler cluster. The first login of a new user triggers a process that sends a verification code to the users ETH email address (USERNAME@ethz.ch, with USERNAME being the ETH account name). The user is then prompted to enter the verification code and by entering the correct code, the cluster account of the user is created.
Leonhard
Leonhard Open (2017-2021) has been integrated in the Euler cluster.
Access to Leonhard Med 1.0 is restricted to Leonhard Med 1.0 shareholders. Guest users cannot access the Leonhard cluster.
MATLAB Distributed Computing Server (MDCS)
Any member of ETH can use the MATLAB Distributed Computing Server (MDCS) service; the only requirement is a valid ETH account. In order to use this service, you first need to login to the Euler cluster once and accept the usage agreement.
Please note that the MDCS will be phased out end of 2022 due to transitioning the batch system from IBM LSF to Slurm
CLC Genomics Server
The CLC genomics server uses local accounts for authentication. If you would like to use this service, then please contact cluster support to request your CLC account.
Please note that the CLC Genomics Server will be phased out end of 2022 due to transitioning the batch system from IBM LSF to Slurm
Accessing the clusters
Who can access the HPC clusters
The Euler cluster is open to all members of ETH and external users that have a collaboration with a research group at ETH Zurich. Members of other institutes who have a collaboration with a research group at ETH may use the HPC clusters for the purpose of said collaboration. Their counterpart ("sponsor") at ETH must ask the local IT support group (ISG) of the corresponding department to create an ETH guest account for them. The account needs to be linked to a valid e-mail address. For external users, the VPN service also needs to be enabled. Once the ETH guest account has been created, they can access the clusters like members of the ETH.
Legal compliance
The HPC clusters of ID SIS HPC are subject to ETH's acceptable use policy for IT resources (Benutzungsordnung für Telematik an der ETH Zürich, BOT). In particular:
- Accounts are strictly personal.
- You must not share your account (password, ssh keys) wih anyone else.
- You must not use someone else's account, with our without their consent.
- If you suspect that someone used your account, change your password and contact cluster support.
For changing your ETH password in the Identity and Access Management (IAM) system of ETH, please have a look at the documentation and the video of IT Services.
In case of abuse, the offender's account may be blocked temporarily or closed. System administrators are obliged by law to investigate abusive or illegal activities and report them to the relevant authorities.
Security
Access to the HPC clusters of ID SIS HPC is only possible via secure protocols ( ssh, sftp, scp, rsync). The HPC clusters are only accessible from inside the ETH network. If you would like to connect from a computer, which is not inside the ETH network, then you would need to establish a VPN connection first. Outgoing connections to computers inside the ETH network are not blocked. If you would like to connect to an external service, then please use the ETH proxy service (http://proxy.ethz.ch:3128) by loading the eth_proxy module:
module load eth_proxy
WARNING: the ETH proxy service can only handle a small number of requests at the same time and its bandwidth is limited. It should therefore be used with moderation. If you have jobs that rely on the ETH proxy to access some external server, please only execute a small number of such jobs at the same time. If you fail to follow this rule, the proxy may completely block access to that external server! This block will affect not only you, but also all users who need to access this external server.
SSH
You can connect to the HPC clusters via the SSH protocol. For this purpose it is required that you have an SSH client installed. The information required to connect to an HPC cluster, is the hostname of the cluster that you would like to connect to and your ETH account credentials (username, password).
| Cluster | Hostname |
|---|---|
| Euler | euler.ethz.ch |
| Leonhard Open | login.leonhard.ethz.ch |
Linux, Mac OS X
Open a shell (Terminal in OS X) and use the standard ssh command
ssh username@hostname
where username is your ETH username and the hostname can be found in the table shown above. If for instance user sfux would like to access the Euler cluster, then the command would be
samfux@bullvalene:~$ ssh sfux@euler.ethz.ch
sfux@euler.ethz.ch's password:
Last login: Fri Sep 13 07:33:57 2019 from bullvalene.ethz.ch
____________________ ___
/ ________ ___ /__/ /
/ _____/ / / / ___ /
/_______/ /__/ /__/ /__/
Eidgenoessische Technische Hochschule Zuerich
Swiss Federal Institute of Technology Zurich
-------------------------------------------------------------------------
E U L E R C L U S T E R
https://scicomp.ethz.ch
http://www.smartdesk.ethz.ch
cluster-support@id.ethz.ch
=========================================================================
[sfux@eu-login-19-ng ~]$
Windows
Since Windows does not provide an ssh client as part of the operating system, users need to download a third-party software in order to be able to establish ssh connections.
Widely used ssh clients are for instance MobaXterm, PuTTY and Cygwin.
For using MobaXterm, you can either start a local terminal and use the same SSH command as for Linux and Mac OS X, or you can click on the session button, choose SSH and then enter the hostname and username. After clicking on OK, you will be asked to enter your password.
If you use PuTTY, then it is sufficient to specify the hostname of the cluster that you would like to access and to click on the Open button. Afterwards, the users will be prompted to enter their ETH account credentials. When using Cygwin, then you can enter the same command as Linux and Mac OS X users.
ssh username@hostname
SSH keys
ssh keys allow you to login to a cluster without having to type a password. This can be useful for file transfer and automated tasks. When you use ssh keys properly, then this is much safer than passwords. There are always pairs of keys, a private (stored on your local workstation) and a public (stored on the computer you want to connect to). You can generate as many key pairs as you want. In order to make the keys even more secure, you should protect them with a passphrase.
Linux, Mac OS X
For a good documentation on SSH please have a look at the SSH website. It contains a general overview on SSH, instructions on how to create SSH keys and instructions on how to copy an SSH key.
On your computer, use ssh-keygen -t ed25519 to generate a key pair with the ed25519 algorithm. By default the private key is stored as $HOME/.ssh/id_ed25519 and the public key as $HOME/.ssh/id_ed25519.pub.
For security reasons, we recommend that you use a different key pair for every computer you want to connect to. For instance, if you are using both Euler and Leonhard:
ssh-keygen -t ed25519 -f $HOME/.ssh/id_ed25519_euler # please enter a strong, non-empty passphrase when prompted ssh-keygen -t ed25519 -f $HOME/.ssh/id_ed25519_leonhard # please enter a strong, non-empty passphrase when prompted
Once this is done, copy the public key to Euler or Leonhard using one of the commands:
ssh-copy-id -i $HOME/.ssh/id_ed25519_euler.pub username@euler.ethz.ch ssh-copy-id -i $HOME/.ssh/id_ed25519_leonhard.pub username@login.leonhard.ethz.ch
Where username is your ETH username. You will need to enter your ETH (LDAP) password to connect to Euler / Leonhard.
If you use an SSH agent, then you also need to add the key there (https://www.ssh.com/ssh/add).
Windows
For windows a third party software (PuTTYgen,MobaXterm) is required to create SSH keys. For a good documentation on SSH please have a look at the SSH website.
Please either use PuTTYgen or the command (MobaXterm)
ssh-keygen -t ed25519
to generate a key pair with the ed25519 algorithm and store both, the public and the private key on your local computer. For security reasons, we recommend that you use a different key pair for every computer you want to connect to. For instance, if you are using both Euler and Leonhard, then save the keys as id_ed25519_euler.pub and id_ed25519_leonhard.pub.
Afterwards please login to the cluster and create the hidden directory $HOME/.ssh which needs to have the unix permission 700.
mkdir -p -m 700 $HOME/.ssh
In order to setup passwordless access to a cluster, copy the public key from your workstation to the $HOME/.ssh directory on the cluster (for this example, we use the Euler cluster, if you would like to setup access to another cluster, then you need to use the corresponding hostname instead of euler.ethz.ch) using for instance WinSCP or MobaXterm. The file needs to be stored as
$HOME/.ssh/authorized_keys
on the cluster.
Safety rules
- Always use a (strong) passphrase to protect your SSH key. Do not leave it empty!
- Never share your private key with somebody else, or copy it to another computer. It must only be stored on your personal computer
- Use a different key pair for each computer you want to connect to
- Do not reuse the key pairs for Euler / Leonhard for other systems
- Do not keep open SSH connections in detached screen sessions
- Disable the ForwardAgent option in your SSH configuration and do not use ssh -A (or use ssh -a to disable agent forwarding)
How to use keys with non-default names
If you use different key pairs for different computers (as recommended above), you need to specify the right key when you connect, for instance:
ssh -i $HOME/.ssh/id_ed25519_euler username@euler.ethz.ch
To make your life easier, you can configure your ssh client to use this option automatically by adding the following lines in your $HOME/.ssh/config file:
Host login.leonhard.ethz.ch IdentityFile ~/.ssh/id_ed25519_leonhard Host euler.ethz.ch IdentityFile ~/.ssh/id_ed25519_euler
First login
On your first login, you need to accept the cluster's usage rules. Afterwards your account is created automatically. Please find below the user agreement for the Euler cluster as an example:
Please note that the Euler cluster is subject to the "Acceptable Use Policy
for Telematics Resources" ("Benutzungsordnung fuer Telematik", BOT) of ETH
Zurich and relevant documents (https://tinyurl.com/eth-bot), in particular:
* your Euler account (like your ETH account) is *strictly personal*
* you are responsible for all activities done under your account
* you must keep your password secure and may not give it to a 3rd party
* you may not share your account with anyone, including your supervisor
* you may not use someone else's account, with or without their consent
* you must comply with all civil and criminal laws (copyright, privacy,
data protection, etc.)
* any violation of these rules and policies may lead to administrative
and/or legal measures
Before you can proceed you must confirm that you have read, understood,
and agree to the rules and policies mentioned above.
On Euler and Leonhard Open, the first login of a new user (for Leonhard Open only for shareholder users) triggers a process that sends a verification code to the users ETH email address (USERNAME@ethz.ch, with USERNAME being the ETH account name). The user is then prompted to enter the verification code and by entering the correct code, the cluster account of the user is created.
X11
The clusters of ID SIS HPC use the X window System (X11) to display a program's graphical user interface (GUI) on a users workstation. You need to install an X11 server on your workstation to siplay X11 windows. The ports used by X11 are blocked by the cluster's firewall. To circumvent this problem, you must open an SSH tunnel and redirect all X11 communication through that tunnel.
Linux
Xorg (X11) is normally installed by default as part of most Linux distributions. If you are using a version newer than 1.16, then please have a look at the troubleshooting section at the bottom of this wiki page.
ssh -Y username@hostname
Mac OS X
Since X11 is no longer included in OS X, you must install XQuartz. If you are using a version newer than 2.7.8, then please have a look at the troubleshooting section at the bottom of this wiki page.
ssh -Y username@hostname
Windows
X11 is not supported by Windows. You need to install a third-party application in order to use X11 forwarding. You can find a list of common X11 servers below:
VPN
When connecting from outside of the ETH network to one of our HPC clusters, you first need to establish a VPN connection. For installing a VPN client, please access https://sslvpn.ethz.ch in your browser. After logging in to the website, it will detect if there is already a VPN client installed on your computer and otherwise install one automatically. You can find more detailed instructions on the ETH website.
Please note that for establishing a VPN connection, you need to use your network password instead of your main password. If you did not yet set your network password, then please go to https://password.ethz.ch, login with your ETH account credentials and click on Passwort ändern. There you can set your network password.
After establishing a VPN connection, you can login to our clusters via SSH.
Troubleshooting
Permission denied
If you enter 3 times a wrong password, then you will get a permission denied error:
sfux@calculus:~$ ssh sfux@euler.ethz.ch sfux@euler.ethz.ch's password: Permission denied, please try again. sfux@euler.ethz.ch's password: Permission denied, please try again. sfux@euler.ethz.ch's password: Permission denied (publickey,password,hostbased). sfux@calculus:~$
In case you receive a "Permission denied" error, please check if you entered the correct password. If you think that your account has been corrupted, then please contact the service desk of IT services of ETH Zurich.
If you enter a wrong password too many times or in a high frequency, then we might block access to the clusters for your account, because it could be correupted. If you account has been blocked by the HPC group, then please contact cluster support.
Timeout
If you try to login and receive a timeout error, then it is very likely that you tried to connect from outside of the ETH network to one of the HPC clusters.
sfux@calculus:~$ ssh -Y sfux@euler.ethz.ch ssh: connect to host euler.ethz.ch port 22: Connection timed out
Please either connect from the inside of the ETH network, or establish a VPN connection.
setlocale: LC_CTYPE: cannot change locale (UTF-8): No such file or directory
If you are using a Mac, can you please try to comment out the following lines in your /etc/ssh/ssh_config on your workstation:
Host *
SendEnv LANG LC_*
This should solve the problem.
Too many authentication failures
This errors can be triggered if you have more than 6 private SSH keys in your local .ssh directory. In this case specify the SSH key to use and use the IdentitiesOnly=yes option, for example:
sfux@calculus:~$ ssh -i $HOME/.ssh/id_ed25519 -o IdentitiesOnly=yes sfux@euler.ethz.ch
Indirect GLX rendering error
When using an SSH connection with X11 forwarding enabled, newer versions of the Xorg server show an error message, when the graphical user interface of an application is started:
X Error of failed request: BadValue (integer parameter out of range for operation) Major opcode of failed request: 153 (GLX) Minor opcode of failed request: 3 (X_GLXCreateContext) Value in failed request: 0x0 Serial number of failed request: 27 Current serial number in output stream: 30
This error is caused by starting your X11 server without enabling the setting for indirect GLX rendering (iglx), that is required for X11 forwarding. Up to version 1.16 of the Xorg server, the setting iglx, has been enabled by default. With version 1.17, the default has changed from +iglx to -iglx. Now the setting needs to be enabled either in the Xorg configuration file or with a command line setting, when starting the Xorg server manually. For Xquartz versions up to 2.7.8, the iglx setting is enabled by default. If you would like to use XQuartz 2.7.9 or newer, then please make sure that you enable the iglx setting when the X-server is started.
This problem is described in the following article:
https://www.phoronix.com/scan.php?page=news_item&px=Xorg-IGLX-Potential-Bye-Bye
Please find below some links, which address the problem for specific operating systems.
| Operating system | Link |
|---|---|
| Red Hat Enterprise Linux (RHEL) | https://elrepo.org/bugs/view.php?id=610 |
| CentOS | https://www.centos.org/forums/viewtopic.php?t=57409#p244528 |
| Ubuntu | http://askubuntu.com/questions/745135/how-to-enable-indirect-glx-contexts-iglx-in-ubuntu-14-04-lts-with-nvidia-gfx |
| Mac OS X | https://bugs.freedesktop.org/show_bug.cgi?id=96260 |
Data management
Introduction
On our cluster, we provide multiple storage systems, which are optimized for different purposes. Since the available storage space on our clusters is limited and shared between all users, we set quotas in order to prevent single users from filling up an entire storage system with their data.
A summary of general questions about file systems, storage and file transfer can be found in our FAQ. If you have questions or encounter problems with the storage systems provided on our clusters or file transfer, then please contact cluster support.
Personal storage (everyone)
Home
On our clusters, we provide a home directory (folder) for every user that can be used for safe long term storage of important and critical data (program source, script, input file, etc.). It is created on your first login to the cluster and accessible through the path
/cluster/home/username
The path is also saved in the variable $HOME. The permissions are set that only you can access the data in your home directory and no other user. Your home directory is limited to 16 GB and a maximum of 100'000 files and directories (inodes). The content of your home is saved every hour and there is also a nightly backup (tape).
Scratch
We also provide a personal scratch directory (folder) for every user, that can be used for short-term storage of larger amounts of data. It is created, when you access it the first time through the path
/cluster/scratch/username
The path is also saved in the variable $SCRATCH. It is visible (mounted), only when you access it. If you try to access it with a graphical tool, then you need to specify the full path as it is might not visible in the /cluster/scratch top-level directory. Before you use your personal scratch directory, please carefully read the usage rules to avoid misunderstandings. The usage rules can also be displayed directly on the cluster with the following command.
cat $SCRATCH/__USAGE_RULES__
Your personal scratch directory has a disk quota of 2.5 TB and a maximum of 1'000'000 files and directories (inodes). There is no backup for the personal scratch directories and they are purged on a regular basis (see usage rules).
For personal scratch directories, there are two limits (soft and hard quota). When reaching the soft limit (2.5 TB) there is a grace period of one week where users can use 10% more than their allowed capacity (this upper limit is called hard quota), which applies to both, the number of inodes and space. If the used capacity is still above the soft limit after the grace period, then the current directory is locked for new writes until being again below the soft quota.
Project
Shareholder groups have the option to purchase additional storage inside the cluster. The project file system is designed for safe long-term storage of critical data (like the home directory). Shareholder groups can buy as much space as they need. The path for project storage is
/cluster/project/groupname
Access rights and restriction is managed by the shareholder group. We recommend to use ETH groups for this purpose. If you are interested in getting more information and prices of the project storage, then please contact cluster support.
Work
Apart from project storage, shareholder groups also have the option to buy so-called work (high-performance) storage. It is optimized for I/O performance and can be used for short- or medium-term storage for large computations (like scratch, but without regular purge). Shareholders can buy as much space as they need. The path for work storage is
/cluster/work/groupname
Access rights and restriction is managed by the shareholder group. We recommend to use ETH groups for this purpose. The directory is visible (mounted), only when accessed. If you are interested in getting more information and prices of the work storage, then please contact cluster support.
For /cluster/work directories, there are two limits (soft and hard quota). When reaching the soft limit there is a grace period of one week where users can use 10% more than their allowed capacity (this upper limit is called hard quota), which applies to both, the number of inodes and space. If the used capacity is still above the soft limit after the grace period, then the current directory is locked for new writes until being again below the soft quota.
Local scratch (on each compute node)
The compute nodes in our HPC clusters also have some local hard drives, which can be used for temporary storing data during a calculation. The main advantage of the local scratch is, that it is located directly inside the compute nodes and not attached via the network. This is very beneficial for serial, I/O-intensive applications. The path of the local scratch is
/scratch
You can either create a directory in local scratch yourself, as part of a batch job, or you can use a directory in local scratch, which is automatically created by the batch system. LSF creates a unique directory in local scratch for every job. At the end of the job, LSF is also taking care of cleaning up this directory. The path of the directory is stored in the environment variable
$TMPDIR
If you use $TMPDIR, then you need to request scratch space from the batch system.
External storage
Please note that external storage is convenient to bring data in to the cluster or to store data for a longer time. But we recommend to not directly process data from external storage systems in batch jobs on Euler as this could be very slow and potentially put a high load on the external storage system. Please rather copy data from the external storage system to some cluster storage (home directory, personal scratch directory, project storage, work storage, or local scratch) before you process it in a batch job. After processing the data from a cluster storage system, you can copy the results back to the external storage system.
Central NAS/CDS
Groups who have purchased storage on the central NAS of ETH or CDS can ask the storage group of IT services to export it to our HPC clusters. There are certain requirements that need to be fulfilled in order to use central NAS/CDS shares on our HPC clusters.
- The NAS/CDS share needs to be mountable via NFS (shares that only support CIFS cannot be mounted on the HPC clusters).
- The NAS/CDS share needs to be exported to the subnet of our HPC clusters (please contact ID Systemdienste and ask them for an NFS export of your NAS/CDS share).
- Please carefully set the permissions of the files and directories on your NAS/CDS share if other cluster users should not have read/write access to your data.
NAS/CDS shares are then mounted automatically when you access them. The mount-point of such a NAS/CDS share is
/nfs/servername/sharename
A typical NFS export file to export a share to the Euler cluster would look like
# cat /etc/exports /export 129.132.93.64/26(rw,root_squash,secure) 10.205.0.0/16(rw,root_squash,secure) 10.204.0.0/16(rw,root_squash,secure)
If you ask the storage group to export your share to the Euler cluster, then please provide them the above-shown information. If the NAS share is located on the IBM Spectrum Scale storage system, then please also ask for the following options to be set by the storage group:
PriviledgedPort=TRUE Manage_Gids=TRUE
Please note that these options should only be applied to the Euler subnet. For a general overview on subnets and IP addresses please check the following wiki page. When a NAS share is mounted on our HPC clusters, then it is accessible from all the compute nodes in the cluster.
Local NAS
Groups that operate their own NAS, can export a shared file system via NFSv3 to our HPC clusters. In order to use an external NAS on our HPC clusters, the following requirements need to be fullfilled
- NAS needs to support NFSv3 (this is currently the only NFS version that is supported from our side).
- The user and group ID's on the NAS needs to be consistent with ETH user names and group.
- The NAS needs to be exported to the subnet of our HPC clusters.
- Please carefully set the permissions of the files and directories on your NAS share if other cluster users should not have read/write access to your data.
We advise you to not use this path directly from your jobs. Rather, you should stage files to and from $SCRATCH.
You external NAS can then be accessed through the mount-point
/nfs/servername/sharename
A typical NFS export file to export a share to the Euler cluster would look like
# cat /etc/exports /export 129.132.93.64/26(rw,root_squash,secure) 10.205.0.0/16(rw,root_squash,secure) 10.204.0.0/16(rw,root_squash,secure)
For a general overview on subnets and IP addresses please check the following wiki page.
The share is automatically mounted, when accessed.
Central LTS (Euler)
Groups who have purchased storage on the central LTS of ETH can ask the ITS SD backup group to export it to the LTS nodes in the Euler cluster. There are certain requirements that need to be fulfilled in order to use central LTS shares on our HPC clusters.
- The LTS share needs to be mountable via NFS (shares that only support CIFS cannot be mounted on the HPC clusters).
- The LTS share needs to be exported to the LTS nodes of our HPC clusters (please contact ITS SD Backup group and ask them for an NFS export of your LTS share).
- Please carefully set the permissions of the files and directories on your LTS share if other cluster users should not have read/write access to your data.
The LTS share needs to be exported to the LTS nodes:
129.132.93.70(rw,root_squash,secure) 129.132.93.71(rw,root_squash,secure)
For accessing your LTS share, you would need to login to the LTS nodes in Euler with
ssh USERNAME@lts.euler.ethz.ch
Where USERNAME needs to be replaced with your ETH account name. LTS shares are then mounted automatically when you access them. The mount-point of such a LTS share is
/nfs/lts11.ethz.ch/shares/sharename(_repl)
or
/nfs/lts21.ethz.ch/shares/sharename(_repl)
depending if your share is located on lts11.ethz.ch or lts21.ethz.ch.
Backup
The users home directories are backed up every night and the backup has a retention time of 90 days. For project and work storage, we provide a weekly back up, with also 90 days retention time. If you have some data that you would like to exclude from the backup, then please create a subdirectory nobackup. Data stored in the nobackup directory will then be excluded from the backup. The subdirectory nobackup can be located on any level in the directory hierarchy:
/cluster/work/YOUR_STORAGE_SHARE/nobackup /cluster/work/YOUR_STORAGE_SHARE/project101/nobackup /cluster/work/YOUR_STORAGE_SHARE/project101/data/nobackup/filename /cluster/work/YOUR_STORAGE_SHARE/project101/data/nobackup/subdir/filename
When large unimportant temporary data that changes a lot is backed up, then this will increase the size/pool of the backup and hence make the backup and the restore process slower. We would therefore like to ask you to exclude this kind of data from the backup of your group storage share if possible. Excluding large temporary data from the backup will help you and us restoring your important data faster in the case of an event.
Comparison
In the table below, we try to give you an overview of the available storage categories/systems on our HPC clusters as well as a comparison of their features.
| Category | Mount point | Life span | Snapshots | Backup | Retention time of backup | Purged | Max. size | Small files | Large files |
|---|---|---|---|---|---|---|---|---|---|
| Home | /cluster/home | permanent | up to 7 days | yes | 90 days | no | 16 GB | + | o |
| Scratch | /cluster/scratch | 2 weeks | no | no | - | yes (files older than 15 days) | 2.5 TB | o | ++ |
| Project | /cluster/project | 4 years | optional | yes | 90 days | no | flexible | + | + |
| Work | /cluster/work | 4 years | no | yes | 90 days | no | flexible | o | ++ |
| Central NAS | /nfs/servername/sharename | flexible | up to 8 days | yes | 90 days | no | flexible | + | + |
| Local scratch | /scratch | duration of job | no | no | - | end of job | 800 GB | ++ | + |
Choosing the optimal storage system
When working on an HPC cluster that provides different storage categories/systems, the choice of which system to use can have a big influence of the performance of your workflow. In the best case you can speedup your workflow by quite a lot, whereas in the worst case the system administrator has to kill all your jobs and has to limit the number of concurrent jobs that you can run because your jobs slow down the entire storage system and this can affect other users jobs. Please take into account a few recommendations that are listed below.
- Use local scratch whenever possible. With a few exceptions this will give you the best performance in most cases.
- For parallel I/O with large files, the high-performance (work) storage will give you the best performance.
- Don't create a large number of small files (KB's) on project or work storage as this could slow down the entire storage system.
- If your application does very bad I/O (opening and closing files multiple times per second and doing small appends on the order of a few bytes), then please don't use project and work storage. The best option for this use-case would be local scratch.
If you need to work with a large amount of small files, then please keep them grouped in a tar archive. During a job you can then untar the files to the local scratch, process them and group the results again in a tar archive, which can then be copied back to your home/scratch/work/project space.
File transfer
In order to run your jobs on a HPC cluster, you need to transfer some data or input files from/to the cluster. For smaller and medium amounts of data, you can use some standard command line/graphical tools. If you need to transfer very large amounts of data (on the order of several TB), then please contact the cluster support and we will help you to set up the optimal strategy to transfer your data in a reasonable amount of time.
Command line tools
For transferring files from/to the cluster, we recommend to use standard tools like secure copy (scp) or rsync. The general syntax for using scp is
scp [options] source destination
For copying a file from your PC to an HPC cluster (to your home directory), you need to run the following command on your PC:
scp file username@hostname:
Where username is your ETH username and hostname is the hostname of the cluster. Please note the colon after the hostname. For copying a file from the cluster to your PC (current directory), you need to run the following command on your PC:
scp username@hostname:file .
For copying an entire directory, you would need to add the option -r. Therefore you would use the following command to transfer a directory from your PC to an HPC cluster (to your home directory).
scp -r directory username@hostname:
The general sytnax for rsync is
rsync [options] source destination
In order to copy the content of a directory from your PC (home directory) to a cluster (home directory), you would use the following command.
rsync -Pav /home/username/directory/ username@hostname:/cluster/home/username/directory
The -P option enables rsync to show the progress of the file transfer. The -a option preserves almost all file attributes and the -v option gives you more verbose output.
Graphical tools
Graphical scp/sftp clients allow you to mount your Euler home directory on your workstation. These clients are available for most operating systems.
- Linux + Gnome: Connect to server
- Linux + KDE: Konqueror, Dolphin, Filezilla
- Mac OS X: MacFUSE, Macfusion, Cyberduck, Filezilla
- Windows: WinSCP, Filezilla'
WinSCP provides the user a Windows explorer like user interface with a split screen that allows to transfer files per drag-and-drop. After starting your graphical scp/sftp client, you need to specify the hostname of the cluster that you would like to connect to and then click on the connect button. After entering your ETH username and password, you will be connected to the cluster and can transfer files.
| WinSCP | |
|---|---|
 |
|
| Filezilla | |
|---|---|
 |
|
Globus for fast file transfer
see Globus for fast file transfer
Quotas
The home and scratch directories on our clusters are subject to a strict user quota. In your home directory, the soft quota for the amount of storage that you can use is set to 16 GiB (17.18 GB) and the hard quota is set to 20 GiB (21.47 GB). Further more, you can store maximally 100'000 files and directories (inodes). The hard quota for your personal scratch directory is set to 2.5 TB. You can maximally have 1'000'000 files and directories (inodes). You can check your current usage with the lquota command.
[sfux@eu-login-13-ng ~]$ lquota +-----------------------------+-------------+------------------+------------------+------------------+ | Storage location: | Quota type: | Used: | Soft quota: | Hard quota: | +-----------------------------+-------------+------------------+------------------+------------------+ | /cluster/home/sfux | space | 8.85 GB | 17.18 GB | 21.47 GB | | /cluster/home/sfux | files | 25610 | 80000 | 100000 | +-----------------------------+-------------+------------------+------------------+------------------+ | /cluster/shadow | space | 4.10 kB | 2.15 GB | 2.15 GB | | /cluster/shadow | files | 2 | 50000 | 50000 | +-----------------------------+-------------+------------------+------------------+------------------+ | /cluster/scratch/sfux | space | 237.57 kB | 2.50 TB | 2.70 TB | | /cluster/scratch/sfux | files | 29 | 1000000 | 1500000 | +-----------------------------+-------------+------------------+------------------+------------------+ [sfux@eu-login-13-ng ~]$
If you reach 80% of your quota (number of files or storage) in your personal scratch directory, you will be informed via email to clean up.
Shareholders that own storage in /cluster/work or /cluster/project on Euler or Leonhard can check their quota also by using the lquota command:
[sfux@eu-login-11-ng ~]$ lquota /cluster/project/sis +-----------------------------+-------------+------------------+------------------+------------------+ | Storage location: | Quota type: | Used: | Soft quota: | Hard quota: | +-----------------------------+-------------+------------------+------------------+------------------+ | /cluster/project/sis | space[B] | 6.17 TB | - | 10.41 TB | | /cluster/project/sis | files | 1155583 | - | 30721113 | +-----------------------------+-------------+------------------+------------------+------------------+ [sfux@eu-login-11-ng ~]$
[sfux@eu-login-11-ng ~]$ lquota /cluster/work/sis +-----------------------------+-------------+------------------+------------------+------------------+ | Storage location: | Quota type: | Used: | Soft quota: | Hard quota: | +-----------------------------+-------------+------------------+------------------+------------------+ | /cluster/work/sis | space | 8.36 TB | 10.00 TB | 11.00 TB | | /cluster/work/sis | files | 1142478 | 10000000 | 11000000 | +-----------------------------+-------------+------------------+------------------+------------------+ [sfux@eu-login-11-ng ~]$
The lquota script requires the path to the top-level directory as parameter.
Setting up the Environment
Introduction
Most applications, compilers and libraries rely on environment variables to function properly. These variables are usually set by the operating system, the administrator, or by the user. Typical examples include:
- PATH — location of system commands and user programs
- LD_LIBRARY_PATH — location of the dynamic (=shared) libraries needed by these commands and programs
- MANPATH — location of man (=manual) pages for these commands
- Program specific environment variables
The majority of problems encountered by users are caused by incorrect or missing environment variables. People often copy initialization scripts — .profile, .bashrc, .cshrc — from one machine to the next, without verifying that the variables defined in these scripts are correct (or even meaningful!) on the target system.
If setting environment variables is difficult, modifying them at run-time is even more complex and error-prone. Changing the contents of PATH to use a different compiler than the one set by default, for example, is not for the casual user. The situation can quickly become a nightmare when one has to deal with multiple compilers and libraries (e.g. MPI) at the same time.
Environment modules — modules in short — offer an elegant and user-friendly solution to all these problems. Modules allow a user to load all the settings needed by a particular application on demand, and to unload them when they are no longer needed. Switching from one compiler to the other; or between different releases of the same application; or from one MPI library to another can be done in a snap, using just one command — module.
Software stacks
On our clusters we provide multiple software stacks.
- The old software stack on Euler uses environment modules
- The new software stack on Euler and the software stack on Leonahrd Open uses LMOD modules
When you login to Euler, then the old software stack is set as default. You can change to the new software stack using the command
env2lmod
For information regarding the new software stack, please have a look at our wiki page regarding SPACK and LMOD
Module commands
Module avail
The module avail command lists all available modules of the supported module category. If you load the new or the legacy module, it will also list all modules of these categories. It can be used to get a quick overview of all centrally installed software. If you are interested in a particular software and would like to know which versions are available, then you can specify the name of the software as a parameter for the module avail command
[sfux@euler01 ~]$ module avail gcc --------------- /cluster/apps/modules/modulefiles --------------- gcc/4.4.7(4.4) gcc/4.8.2(default) gcc/4.9.2 [sfux@euler01 ~]$ module load legacy new [sfux@euler01 ~]$ module avail gcc --------------- /cluster/apps/modules/modulefiles --------------- gcc/4.4.7(4.4) gcc/4.8.2(default) gcc/4.9.2 ----------------- /cluster/apps/modules/legacy ------------------ gcc/4.7.4 ------------------- /cluster/apps/modules/new ------------------- gcc/4.8.4 gcc/5.2.0
Module show
The module show command provides you some information on what environment variables are changed and set by the module file. Further more it also contains a short string with information about the name and the version of the application or library.
[sfux@euler01 ~]$ module show python/2.7.6 ------------------------------------------------------------------- /cluster/apps/modules/modulefiles/python/2.7.6: module-whatis Python version 2.7.6 (x86_64) prepend-path PATH /cluster/apps/python/2.7.6/x86_64/bin prepend-path LD_LIBRARY_PATH /cluster/apps/python/2.7.6/x86_64/lib64 prepend-path PKG_CONFIG_PATH /cluster/apps/python/2.7.6/x86_64/lib64/pkgconfig setenv PYTHON_ROOT /cluster/apps/python/2.7.6/x86_64 -------------------------------------------------------------------
Module load
The module load command load the corresponding and prepares the environment for using this application or library, by applying the instructions, which can be shown by running the module show command.
[sfux@euler01 ~]$ module load gcc/4.8.2 python/2.7.6 Autoloading openblas/0.2.13_seq [sfux@euler01 ~]$ which python /cluster/apps/python/2.7.6/x86_64/bin/python
Module list
The module list command displays the currently loaded modules files.
[sfux@euler04 ~]$ module list Currently Loaded Modulefiles: 1) modules [sfux@euler04 ~]$ module load gcc/4.8.2 python/2.7.6 Autoloading openblas/0.2.13_seq [sfux@euler04 ~]$ module list Currently Loaded Modulefiles: 1) modules 3) openblas/0.2.13_seq(default:seq) 2) gcc/4.8.2(default:4.8) 4) python/2.7.6(2.7)
Module purge
The module purge command unload all currently loaded modules and cleans up the environment of your shell. In some cases, it might be better to log out and log in again, in order to get a really clean shell.
[sfux@euler04 ~]$ module list Currently Loaded Modulefiles: 1) modules 3) openblas/0.2.13_seq(default:seq) 2) gcc/4.8.2(default:4.8) 4) python/2.7.6(2.7) [sfux@euler04 ~]$ module purge [sfux@euler04 ~]$ module list No Modulefiles Currently Loaded.
Naming scheme
Please find the general naming scheme of module files below.
program_name/version(alias[:alias2])
Instead of specifying a version directly, it is also possible to use aliases.
program_name/alias == program_name/version
The special alias default indicates which version is taken by default (if neither version nor alias is specified)
program_name/default == program_name
If no default is specified for a particular software, then the most recent version (i.e. that with the largest number) is taken by default.
LMOD
For the Leonhard cluster and the new software stack on Euler, we decided to switch from the environment modules that are used for the old software stack on Euler to Lmod modules, which provide some additional features. You should barely notice the transition from environment modules to Lmod modules as the commands are mostly the same. Therefore please refer to the Setting up your environment tutorial for a general documentation about the module commands.
[sfux@lo-login-02 ~]$ module avail boost ----------------------------------------- /cluster/apps/lmodules/Compiler/gcc/4.8.5 ------------------------------------------ boost/1.63.0 Use "module spider" to find all possible modules. Use "module keyword key1 key2 ..." to search for all possible modules matching any of the "keys". [sfux@lo-login-02 ~]$ module load boost/1.63.0 [sfux@lo-login-02 ~]$ module list Currently Loaded Modules: 1) gcc/4.8.5 2) StdEnv 3) boost/1.63.0 [sfux@lo-login-02 ~]$
Please note that this is work in progress and the module names might change. Currently, the number of software packages provided on Leonhard is not comparable to the software we provide on the Euler cluster, but it will grow over time.
Hierarchical modules
LMOD allows to define a hierarchy of modules containing 3 layers (Core, Compiler, MPI). The core layer contains all module files which are not depending on any compiler/MPI. The compiler layer contains all modules which are depending on a particular compilers, but not on any MPI library. The MPI layer contains modules that are depending on a particular compiler/MPI combination.
When you login to the Leonhard cluster, the standard compiler gcc/4.8.5 is automatically loaded. Running the module avail command displays all modules that are available for gcc/4.8.5. If you would like to see the modules available for a different compiler, for instance gcc/6.3.0, then you would need to load the compiler module and run module avail again. For checking out the available modules for gcc/4.8.5 openmpi/2.1.0, you would load the corresponding compiler and MPI module and run again module avail'.
As a consequence of the module hierarchy, you can never have two different versions of the same module loaded at the same time. This helps to avoid problems arising due to misconfiguration of the environment.
Application life-cycle
Based on application experience on Brutus we offer, besides the currently supported versions, two new categories of modules for new and legacy versions on the Euler cluster. Due to dependencies between compilers, libraries and applications, changes to the applications and the corresponding modules need to be synchronized.
Life-cycle of an application
Modules for new or experimental versions of an application/library/compiler first appear in the new module category, where we provide a partial support matrix. Specific compiler/library combinations can be requested by shareholders. New modules are not visible by default. If you would like to see which new versions are available or try them out, you will need to load the new module first:
module load new
Applications that have passed all tests and are deemed ready for production (stable, bug-free, compatible with LSF, etc.) will be moved to the supported category in the next quarterly update.
Applications that have become obsolete (no longer supported by the vendor, superseded by new versions with more functionality, etc.) will be moved to the legacy category. For these modules the HPC group can only provide limited support. Legacy modules are not visible by default. If you still need to use them, you will need to load the legacy module first:
module load legacy
Applications that are known to be buggy, have become redundant or whose license have expired will be removed. If you still need to use them, please contact cluster support.
User notification
The HPC group updates the module categories on a regular basis. Therefore the Application life-cycle page contains a table listing all applications that are available on Euler as well as the modifications that we plan to apply at the next change. The users will receive a reminder, one week prior to the update of the categories that will also contain information about the most important changes.
Application tables
We have listed all available modules on the different HPC clusters in separate tables, which also contain a special formatting that indicate actions taken in the next change. Please note that the application table for Leonhard does not contain the different module categories as we have switched from manual installations to using the supercomputing package manager (SPACK) and also switched from environment modules to lmod modules.
Using the batch system
Introduction
On our HPC cluster, we use the IBM LSF (Load Sharing Facility) batch system. A basic knowledge of LSF is required if you would like to work on the HPC clusters. The present article will show you how to use LSF to execute simple batch jobs and give you an overview of some advanced features that can dramatically increase your productivity on a cluster.
Using a batch system has numerous advantages:
- single system image — all computing resources in the cluster can be accessed from a single point
- load balancing — the workload is automatically distributed across all available processor cores
- exclusive use — many computations can be executed at the same time without affecting each other
- prioritization — computing resources can be dedicated to specific applications or people
- fair share — a fair allocation of those resources among all users is guaranteed
In fact, our HPC clusters contains so many cores (70,000) and are used by so many people (more than 3,000) that it would be impossible to use it efficiently without a batch system.
All computations on our HPC cluster must be submitted to the batch system. Please do not run any job interactively on the login nodes, except for testing or debugging purposes.
If you are a member of multiple shareholder groups, then please have a look at our wiki page about working in multiple shareholder groups
Basic job submission
Simple commands and programs
Submitting a job to the batch system is as easy as:
bsub command [arguments] bsub /path/to/program [arguments]
Examples:
[sfux@euler03 ~]$ bsub gzip big_file.dat Generic job. Job <8146539> is submitted to queue <normal.4h>.
[sfux@euler03 ~]$ bsub ./hello_world Generic job. Job <8146540> is submitted to queue <normal.4h>.
Two or more commands can be combined together by enclosing them in quotes:
bsub "command1; command2"
Example:
[sfux@euler03 ~]$ bsub "configure; make; make install" Generic job. Job <8146541> is submitted to queue <normal.4h>.
Quotes are also necessary if you want to use I/O redirection (">", "<"), pipes ("|") or conditional operators ("&&", "||"):
bsub "command < data.in > data.out" bsub "command1 | command2"
Examples:
[sfux@euler03 ~]$ bsub "tr ',' '\n' < comma_separated_list > linebreak_separated_list" Generic job. Job <8146542> is submitted to queue <normal.4h>.
[sfux@euler03 ~]$ bsub "cat unsorted_list_with_redundant_entries | sort | uniq > sorted_list" Generic job. Job <8146543> is submitted to queue <normal.4h>.
Shell scripts
More complex commands may be placed in a shell script, which should then be submitted like this:
bsub < script
Example:
[sfux@euler03 ~]$ bsub < hello.sh Generic job. Job <8146544> is submitted to queue <normal.4h>.
In principle, it is also possible to submit a script as if it were a program:
bsub /path/to/script ← BAD IDEA!
however this syntax is strongly discouraged on our clusters because it does not allow the batch system to "see" what your script is doing, which may lead to errors in the submission and/or execution of your job.
Output file
By default your job's output (or standard output, to be precise) is written into a file named lsf.oJobID in the directory where you executed bsub, where JobID is the number assigned to your job by LSF. You can select a different output file using the option:
bsub -o output_file command [argument]
The option -o output_file tells LSF to append your job's output to output_file. If you want to overwrite this file, use:
bsub -oo output_file ...
Note that this option, like all bsub options, must be placed before the command that you want to execute in your job. A common mistake is to place bsub options in the wrong place, like.
bsub command -o output_file ← WRONG!
Batch interactive job
If you just want to run a quick test, you can submit it as a batch interactive job. In this case the job's output is not written into a file, but directly to your terminal, as if it were executed interactively:
bsub -I command [arguments]
Example:
[sfux@euler03 ~]$ bsub -I "env | sort" Generic job. Job <8146545> is submitted to queue <normal.4h>. <<Waiting for dispatch ...>
There are two options -p and -s that can be combined with the -I option.
bsub -Ip
will give you a pseudo-terminal and
bsub -Is
will give you a login shell.
Resource requirements
By default, a batch job can use only one core for up to 4 hours. (The job is killed when it reaches its run-time limit.) If your job needs more resources — time, cores, memory or scratch space —, you must request them when you submit it.
Wall-clock time
The time limits on our clusters are always based on wall-clock (or elapsed) time. You can specify the amount of time needed by your job using the option:
bsub -W minutes ... example: bsub -W 90 ... bsub -W HH:MM ... example: bsub -W 1:30 ...
Examples:
[sfux@euler03 ~]$ bsub -W 20 ./Riemann_zeta -arg 26 Generic job. Job <8146546> is submitted to queue <normal.4h>.
[sfux@euler03 ~]$ bsub -W 20:00 ./solve_Koenigsberg_bridge_problem Generic job. Job <8146547> is submitted to queue <normal.24h>.
Since our clusters contains processor cores with different speeds two similar jobs will not necessarily take the same time to complete. It is therefore safer to request more time than strictly necessary... but not too much, for shorter jobs have generally a higher priority than longer ones.
The maximum run-time for jobs that can run on most compute nodes in the cluster is 360 hours. We remain the right to stop jobs with a run time of more than 5 days in case of an emergency maintenance.
Number of processor cores
If your job requires multiple cores (or threads), you must request them using the option:
bsub -n number_of_procs ...
Note that merely requesting multiple cores does not mean that your application will use them.
Memory
By default the batch system allocates 1024 MB (1 GB) of memory per processor core. A single-core job will thus get 1 GB of memory; a 4-core job will get 4 GB; and a 16-core job, 16 GB. If your computation requires more memory, you must request it when you submit your job:
bsub -R "rusage[mem=XXX]" ...
Example:
[sfux@euler03 ~]$ bsub -R "rusage[mem=2048]" ./evaluate_gamma -precision 10e-30 Generic job. Job <8146548> is submitted to queue <normal.4h>.
where XXX is the amount of memory needed by your job, in MB per processor core.
Scratch space
LSF automatically creates a local scratch directory when your job starts and deletes it when the job ends. This directory has a unique name, which is passed to your job via the variable $TMPDIR.
Unlike memory, the batch system does not reserve any disk space for this scratch directory by default. If your job is expected to write large amounts of temporary data (say, more than 250 MB) into $TMPDIR — or anywhere in the local /scratch file system — you must request enough scratch space when you submit it:
bsub -R "rusage[scratch=YYY]" ...
Example:
[sfux@euler03 ~]$ bsub -R "rusage[scratch=5000]" ./generating_Euler_numbers -num 5000000 Generic job. Job <8146548> is submitted to queue <normal.4h>.
where YYY is the amount of scratch space needed by your job, in MB per processor core.
Note that /tmp is reserved for the operating system. Do not write temporary data there! You should either use the directory created by LSF ($TMPDIR) or create your own temporary directory in the local /scratch file system; in the latter case, do not forget to delete this directory at the end of your job.
Multiple requirements
It is possible to combine memory and scratch requirements:
bsub -R "rusage[mem=XXX]" -R "rusage[scratch=YYY]" ...
is equivalent to:
bsub -R "rusage[mem=XXX,scratch=YYY]" ...
LSF submission line advisor
For users that are not yet very experienced with using a batch system, we provide a small helper tool, which simplifies to setup the command for requesting resources from the batch system in order to submit a job.
https://scicomp.ethz.ch/lsf_submission_line_advisor
GPU
There are GPU nodes in both, the Euler and the Leonhard Open cluster. The GPU nodes on both cluster are reserved exclusively to the shareholder groups that invested into them. Guest users and shareholder that purchase CPU nodes but no GPU nodes cannot use the GPU nodes.
All GPUs are configured in Exclusive Process mode. Please find below the available GPU node types.
Euler
| GPU Model | Specifier (GPU driver <= 450.80.02) | Specifier (GPU driver > 450.80.02) | GPU memory per GPU | CPU cores per node | CPU memory per node |
|---|---|---|---|---|---|
| NVIDIA GeForce GTX 1080 | |
NVIDIAGeForceGTX1080 | 8 GiB | 20 | 256 GiB |
| NVIDIA GeForce GTX 1080 Ti | |
NVIDIAGeForceGTX1080Ti | 11 GiB | 20 | 256 GiB |
| NVIDIA GeForce RTX 2080 Ti | |
NVIDIAGeForceRTX2080Ti | 11 GiB | 36 | 384 GiB |
| NVIDIA GeForce RTX 2080 Ti | |
NVIDIAGeForceRTX2080Ti | 11 GiB | 128 | 512 GiB |
| NVIDIA GeForce RTX 3090 | NVIDIAGeForceRTX3090 | 24 GiB | 128 | 512 GiB | |
| NVIDIA TITAN RTX | |
NVIDIATITANRTX | 24 GiB | 128 | 512 GiB |
| NVIDIA Quadro RTX 6000 | |
QuadroRTX6000 | 24 GiB | 128 | 512 GiB |
| NVIDIA Tesla V100-SXM2 32 GB | |
TeslaV100_SXM2_32GB | 32 GiB | 48 | 768 GiB |
| NVIDIA Tesla V100-SXM2 32 GB | |
TeslaV100_SXM2_32GB | 32 GiB | 40 | 512 GiB |
| Nvidia Tesla A100 | |
NVIDIAA100_PCIE_40GB | 40 GiB | 48 | 768 GiB |
Please note that the update of the GPU driver is a rolling update. For GPU node types where all nodes have already the updated driver version, the old identifier is crossed-out in the table above. Don't use crossed-out identifiers, as your job will be pending forever as LSF cannot find nodes with GPUs that have those identifiers.
To run multi-node job, you will need to request span[ptile=XX] with XX being the number of CPU cores per GPU node, which is depending on the node type (the node types are listed in the table above).
The LSF batch system has partial integrated support for GPUs. To use the GPUs for a job node you need to request the ngpus_excl_p resource. It refers to the number of GPUs per node. This is unlike other resources, which are requested per core.
For example, to run a serial job with one GPU,
bsub -R "rusage[ngpus_excl_p=1]" ./my_cuda_program
or on a full node with all 8 GeForce GTX 1080 Ti GPUs and up to 90 GB of RAM,
bsub -n 20 -R "rusage[mem=4500,ngpus_excl_p=8]" -R "select[gpu_model0==GeForceGTX1080Ti]" ./my_cuda_program
or on two full nodes:
bsub -n 40 -R "rusage[mem=4500,ngpus_excl_p=8] -R "select[gpu_model0==GeForceGTX1080Ti]" span[ptile=20]" ./my_cuda_program
While your jobs will see all GPUs, LSF will set the CUDA_VISIBLE_DEVICES environment variable, which is honored by CUDA programs.
For advanced settings, please have a look at our getting started with GPUs page.
Parallel job submission
Before submitting parallel jobs, please make sure that your application can run in parallel at all in order to not waste resources by requesting multiple cores for a serial application. Further more, please do a short scaling analysis to see how well your code scales in parallel before requesting dozens or hundreds of cores.
OpenMP
If your application is parallelized using OpenMP or linked against a library using OpenMP (Intel MKL, OpenBLAS, etc.), the number of processor cores (or threads) that it can use is controlled by the environment variable OMP_NUM_THREADS. This variable must be set before you submit your job:
export OMP_NUM_THREADS=number_of_cores bsub -R "span[ptile=number_of_cores]" -n number_of_cores ...
NOTE: if OMP_NUM_THREADS is not set, your application will either use one core only, or will attempt to use all cores that it can find, stealing them from other jobs if needed. In other words, your job will either use too few or too many cores.
MPI
Three kinds of MPI libraries are available on our cluster: Open MPI (recommended), Intel MPI and MVAPICH2. Before you can submit and execute an MPI job, you must load the corresponding modules (compiler + MPI, in that order):
module load compiler module load mpi_library
The command used to launch an MPI application is mpirun.
Let's assume for example that hello_world was compiled with PGI 15.1 and linked with Open MPI 1.6.5. The command to execute this job on 4 cores is:
module load pgi/15.1 module load open_mpi/1.6.5 bsub -n 4 mpirun ./hello_world
Note that mpirun automatically uses all cores allocated to the job by LSF. It is therefore not necessary to indicate this number again to the mpirun command itself:
bsub -n 4 mpirun -np 4 ./hello_world ← "-np 4" not needed!
- Open MPI
- Open MPI 1.6.5 has been tested to work with acceptable performance.
- Open MPI 2.0.2 has been tested to work
- MVAPICH2
- MVAPICH2 2.1 works but preliminary results show low scalability. You need to load the interconnect/ethernet module.
- Intel MPI
- Intel MPI 5.1.3 has been tested.
Pthreads and other threaded applications
Their behavior is similar to OpenMP applications. It is important to limit the number of threads that the application spawns. There is no standard way to do this, so be sure to check the application's documentation on how to do this. Usually a program supports at least one of four ways to limit itself to N threads:
- it understands the OMP_NUM_THREADS=N environment variable,
- it has its own environment variable, such as GMX_NUM_THREADS=N for Gromacs,
- it has a command-line option, such as -nt N (for Gromacs), or
- it has an input-file option, such as num_threads N.
If you are unsure about the program's behavior, please contact us and we will analyze it.
Hybrid jobs
It is possible to run hybrid jobs that mix MPI and OpenMP on our HPC clusters, but we strongly recommend to not submit these kind of jobs.
Job monitoring
Please find below a table with commands for job monitoring and job control
| Command | Description |
|---|---|
| busers | user limits, number of pending and running jobs |
| bqueues | queues status (open/closed; active/inactive) |
| bjobs | more or less detailed information about pending, running and recently finished jobs |
| bbjobs | better bjobs (bjobs with human readable output) |
| bhist | information about jobs that finished in the last hours/days |
| bpeek | display the standard output of a given job |
| lsf_load | show the CPU load of all nodes used by a job |
| bjob_connect | login to a node where one of your jobs is running |
| bkill | kill a job |
For an overview on the most common options for the LSF commands, please have a look at the LSF mini reference.
bjobs
The bjobs command allows you to get information about pending, running and shortly finished jobs.
bbjobs
The command bbjobs can be used to see the resource request and usage (cpu, memory, swap, etc.) of any specific job.
bbjobs [-u username -r -a -s -d -p -f -l -P] JOBID
| Option | Description |
|---|---|
| (no option) | List your jobs — information, requested resources and usage. |
| -u username | user username. |
| -r | Show only running jobs. |
| -a | Show all jobs. |
| -s | Show only suspended jobs. |
| -d | Show only jobs that ended recently (done). |
| -p | Show only pending jobs. |
| -f | Show job cpu affinity, which cores it is running. |
| -l | Show job information in log format. |
Example of output for bbjobs:
[sfux@euler08 ~]$ bbjobs 31989961 Job information Job ID : 31989961 Status : RUNNING Running on node : e1268 User : sfux Queue : normal.4h Command : compute_pq.py Working directory : $HOME/testruns Requested resources Requested cores : 1 Requested memory : 1024 MB per core Requested scratch : not specified Dependency : - Job history Submitted at : 08:45 2016-11-15 Started at : 08:48 2016-11-15 Queue wait time : 140 sec Resource usage Updated at : 08:48 2016-11-15 Wall-clock : 34 sec Tasks : 4 Total CPU time : 5 sec CPU utilization : 80.0 % Sys/Kernel time : 0.0 % Total resident memory : 2 MB Resident memory utilization : 0.2 %
bjob_connect
Sometimes it is necessary to monitor the job on the node(s) where it is running. On Euler, compute nodes can not be accessed directly via ssh. To access a node where a job is running the tool bjob_connect should be used.
bjob_connect JOBID [SSH OPTIONS]
The tool will connect directly to the node where the job is running. In the case of multi-node runs, a list of nodes will be printed and one should be chosen to be accessed.
Connections to nodes created via bjob_connect must be ended explicitly (exit from terminal) by the user when done with job monitoring.
Troubleshooting
Bsub rejects my job
If the error message is not self-explanatory, then please report it to the cluster support.
My job is stick in the queue since XXX hours/days
Please try to find out, why the job is pending. You can do this with the following command.
bjobs -p
Individual host-based reasons means that the resources requested by your jobs are not available at this time. Some resources may never become available ( e.g. mem=10000000). Some resource requirements may be mutually exclusive.
My job was sent to the purgatory queue
The purgatory queue is designed to catch jobs that were not submitted properly, either due to a user error or a bug in the batch system. Please always report this type of problem to the cluster support.
Applications
We provide a wide range of centrally installed commercial and open source applications and libraries to our cluster users.
Central installations
Applications and libraries that are used by many people from different departments of ETH (e.g. MATLAB, Comsol, Ansys, etc.) or that are explicitly requested by a shareholder group will be installed centrally in /cluster/apps. Providing a software stack of centrally installed applications and libraries gives the users certain advantages.
- Applications and libraries are visible and accessible to all users via environment modules.
- They are maintained by the ID SIS HPC group (or in a few cases also by users).
- Commercial licenses are provided by the central license administration of ETH (IT shop).
If an application or library is only used by a few people, then we recommend the users to install it locally in their home directory. In case you need help to install an application or library locally, please do not hesitate to contact cluster support.
Software stacks
On our clusters we provide multiple software stacks.
- The old software stack on Euler uses environment modules
- The new software stack on Euler and the software stack on Leonahrd Open uses LMOD modules
On the Euler cluster, we are in the transition phase from the old software stack to the new software stack. Currently when users login, the old software stack is still set as the default. It will be kept on an as-is basis to allow reproducing older results. New software is only installed in the new software stack. It is set up with the package manager SPACK on Euler using the LMOD module system. The new software stack has 4 basic toolchains (GCC 4.8.5, GCC 6.3.0, GCC 8.2.0 and Intel 18.0.1). A toolchain is a combination of compiler, MPI and BLAS/LAPACK library. For each of the toolchains, several hundred packages are available.
For an overview on all installed packages, please have a look at the following wiki pages:
Commercial/Open source
Please find below a selection of centrally installed applications that covers a wide range of research fields
| Bioinformatics |
|---|
| Bamtools, BLAST, Bowtie, CLC Genomics Server, RAxML, Relion, TopHat |
| Finite element methods |
| Ansys, Abaqus, FEniCS |
| Machine learning |
| PyTorch, Scikit-Learn, TensorFlow, Theano |
| Multi-physics phenomena |
| Ansys EDT, COMSOL Multiphysics, STAR-CCM+ |
| Quantum chemistry and molecular dynamics |
| ADF, Ambertools, Gaussian, Molcas, Octopus, Orca, Qbox, Turbomole |
| Symbolic, numberical and statistical mathematics |
| Gurobi, Mathematica, MATLAB, R, Stata |
| Visualization |
| Ffmpeg, ParaView, VisIT, VTK |
Development
We also provide a range of programs or libraries for the development of software, in order to allow users to write their own codes and to run and profile them on the cluster.
| Compiler |
|---|
| GCC, Intel |
| Programming languages |
| C, C++, Fortran, Go, Java, Julia, Perl, Python, Ruby, Scala |
| Scientific libraries |
| Boost, Eigen, FFTW, GMP, GSL, HDF5, MKL, NetCDF, NumPy, OpenBLAS, SciPy |
| MPI libraries |
| Open MPI, Intel MPI, MPICH |
| Build systems |
| GNU Autotools, CMake, QMake, Make, SCons |
| Version control |
| SVN, Git, Mercurial, CVS |
