Accessing the clusters
Contents
Who can access the HPC clusters
The Euler cluster is open to all members of ETH and external users that have a collaboration with a research group at ETH Zurich. Members of other institutes who have a collaboration with a research group at ETH may use the HPC clusters for the purpose of said collaboration. Their counterpart ("sponsor") at ETH must ask the local IT support group (ISG) of the corresponding department to create an ETH guest account for them. The account needs to be linked to a valid e-mail address. For external users, the VPN service also needs to be enabled. Once the ETH guest account has been created, they can access the clusters like members of the ETH.
Legal compliance
The HPC clusters of ID SIS HPC are subject to ETH's acceptable use policy for IT resources (Benutzungsordnung für Telematik an der ETH Zürich, BOT). In particular:
- Accounts are strictly personal.
- You must not share your account (password, ssh keys) wih anyone else.
- You must not use someone else's account, with our without their consent.
- If you suspect that someone used your account, change your password and contact cluster support.
For changing your ETH password in the Identity and Access Management (IAM) system of ETH, please have a look at the documentation and the video of IT Services.
In case of abuse, the offender's account may be blocked temporarily or closed. System administrators are obliged by law to investigate abusive or illegal activities and report them to the relevant authorities.
Internet Security
Access to the HPC clusters of ID SIS HPC is only possible via secure protocols ( ssh, sftp, scp, rsync). The HPC clusters are only accessible from inside the ETH network. If you would like to connect from a computer, which is not inside the ETH network, then you would need to establish a VPN connection first. Outgoing connections to computers inside the ETH network are not blocked. If you would like to connect to an external service, then please use the ETH proxy service (http://proxy.service.consul:3128) by loading the eth_proxy module:
module load eth_proxy
WARNING: the ETH proxy service can only handle a small number of requests at the same time and its bandwidth is limited. It should therefore be used with moderation. If you have jobs that rely on the ETH proxy to access some external server, please only execute a small number of such jobs at the same time. If you fail to follow this rule, the proxy may completely block access to that external server! This block will affect not only you, but also all users who need to access this external server.
SSH
You can connect to the HPC clusters via the SSH protocol. For this purpose it is required that you have an SSH client installed. The information required to connect to an HPC cluster, is the hostname of the cluster that you would like to connect to and your ETH account credentials (username, password).
| Cluster | Hostname |
|---|---|
| Euler | euler.ethz.ch |
Linux, Mac OS X
Open a shell (Terminal in OS X) and use the standard ssh command
ssh username@hostname
where username is your ETH username and the hostname can be found in the table shown above. If for instance user sfux would like to access the Euler cluster, then the command would be
samfux@bullvalene:~$ ssh sfux@euler.ethz.ch
sfux@euler.ethz.ch's password:
Last login: Fri Sep 13 07:33:57 2019 from bullvalene.ethz.ch
____________________ ___
/ ________ ___ /__/ /
/ _____/ / / / ___ /
/_______/ /__/ /__/ /__/
Eidgenoessische Technische Hochschule Zuerich
Swiss Federal Institute of Technology Zurich
-------------------------------------------------------------------------
E U L E R C L U S T E R
https://scicomp.ethz.ch
http://www.smartdesk.ethz.ch
cluster-support@id.ethz.ch
=========================================================================
[sfux@eu-login-19 ~]$
Windows
Since Windows does not provide an ssh client as part of the operating system, users need to download a third-party software in order to be able to establish ssh connections.
Widely used ssh clients are for instance MobaXterm, PuTTY and Cygwin.
For using MobaXterm, you can either start a local terminal and use the same SSH command as for Linux and Mac OS X, or you can click on the session button, choose SSH and then enter the hostname and username. After clicking on OK, you will be asked to enter your password.
If you use PuTTY, then it is sufficient to specify the hostname of the cluster that you would like to access and to click on the Open button. Afterwards, the users will be prompted to enter their ETH account credentials. When using Cygwin, then you can enter the same command as Linux and Mac OS X users.
ssh username@hostname
SSH keys
ssh keys allow you to login to a cluster without having to type a password. This can be useful for file transfer and automated tasks. When you use ssh keys properly, then this is much safer than passwords. There are always pairs of keys, a private (stored on your local workstation) and a public (stored on the computer you want to connect to). You can generate as many key pairs as you want. In order to make the keys even more secure, you should protect them with a passphrase.
Linux, Mac OS X
For a good documentation on SSH please have a look at the SSH website. It contains a general overview on SSH, instructions on how to create SSH keys and instructions on how to copy an SSH key.
On your computer, use ssh-keygen -t ed25519 to generate a key pair with the ed25519 algorithm. By default the private key is stored as $HOME/.ssh/id_ed25519 and the public key as $HOME/.ssh/id_ed25519.pub.
For security reasons, we recommend that you use a different key pair for every computer you want to connect to. For instance, if you are using both Euler and Leonhard:
ssh-keygen -t ed25519 -f $HOME/.ssh/id_ed25519_euler # please enter a strong, non-empty passphrase when prompted
Once this is done, copy the public key to Euler or Leonhard using one of the commands:
ssh-copy-id -i $HOME/.ssh/id_ed25519_euler.pub username@euler.ethz.ch
Where username is your ETH username. You will need to enter your ETH (LDAP) password to connect to Euler / Leonhard.
If you use an SSH agent, then you also need to add the key there (https://www.ssh.com/ssh/add).
Windows
For windows a third party software (PuTTYgen,MobaXterm) is required to create SSH keys. For a good documentation on SSH please have a look at the SSH website.
Please either use PuTTYgen or the command (MobaXterm)
ssh-keygen -t ed25519
to generate a key pair with the ed25519 algorithm and store both, the public and the private key on your local computer. For security reasons, we recommend that you use a different key pair for every computer you want to connect to.
Afterwards please login to the cluster and create the hidden directory $HOME/.ssh which needs to have the unix permission 700.
mkdir -p -m 700 $HOME/.ssh
In order to setup passwordless access to a cluster, copy the public key from your workstation to the $HOME/.ssh directory on the cluster (for this example, we use the Euler cluster, if you would like to setup access to another cluster, then you need to use the corresponding hostname instead of euler.ethz.ch) using for instance WinSCP or MobaXterm. The file needs to be stored as
$HOME/.ssh/authorized_keys
on the cluster.
Safety rules
- Always use a (strong) passphrase to protect your SSH key. Do not leave it empty!
- Never share your private key with somebody else, or copy it to another computer. It must only be stored on your personal computer
- Use a different key pair for each computer you want to connect to
- Do not reuse the key pairs for Euler / Leonhard for other systems
- Do not keep open SSH connections in detached screen sessions
- Disable the ForwardAgent option in your SSH configuration and do not use ssh -A (or use ssh -a to disable agent forwarding)
How to use keys with non-default names
If you use different key pairs for different computers (as recommended above), you need to specify the right key when you connect, for instance:
ssh -i $HOME/.ssh/id_ed25519_euler username@euler.ethz.ch
To make your life easier, you can configure your ssh client to use this option automatically by adding the following lines in your $HOME/.ssh/config file:
Host euler.ethz.ch IdentityFile ~/.ssh/id_ed25519_euler
First login
On your first login, you need to accept the cluster's usage rules. Afterwards your account is created automatically. Please find below the user agreement for the Euler cluster as an example:
Please note that the Euler cluster is subject to the "Acceptable Use Policy
for Telematics Resources" ("Benutzungsordnung fuer Telematik", BOT) of ETH
Zurich and relevant documents (https://tinyurl.com/eth-bot), in particular:
* your Euler account (like your ETH account) is *strictly personal*
* you are responsible for all activities done under your account
* you must keep your password secure and may not give it to a 3rd party
* you may not share your account with anyone, including your supervisor
* you may not use someone else's account, with or without their consent
* you must comply with all civil and criminal laws (copyright, privacy,
data protection, etc.)
* any violation of these rules and policies may lead to administrative
and/or legal measures
Before you can proceed you must confirm that you have read, understood,
and agree to the rules and policies mentioned above.
On Euler, the first login of a new user triggers a process that sends a verification code to the users ETH email address (USERNAME@ethz.ch, with USERNAME being the ETH account name). The user is then prompted to enter the verification code and by entering the correct code, the cluster account of the user is created.
X11
The clusters of ID SIS HPC use the X window System (X11) to display a program's graphical user interface (GUI) on a users workstation. You need to install an X11 server on your workstation to siplay X11 windows. The ports used by X11 are blocked by the cluster's firewall. To circumvent this problem, you must open an SSH tunnel and redirect all X11 communication through that tunnel.
Linux
Xorg (X11) is normally installed by default as part of most Linux distributions. If you are using a version newer than 1.16, then please have a look at the troubleshooting section at the bottom of this wiki page.
ssh -Y username@hostname
Mac OS X
Since X11 is no longer included in OS X, you must install XQuartz. If you are using a version newer than 2.7.8, then please have a look at the troubleshooting section at the bottom of this wiki page.
ssh -Y username@hostname
Windows
X11 is not supported by Windows. You need to install a third-party application in order to use X11 forwarding. You can find a list of common X11 servers below:
VPN
When connecting from outside of the ETH network to one of our HPC clusters, you first need to establish a VPN connection. For installing a VPN client, please access https://sslvpn.ethz.ch in your browser. After logging in to the website, it will detect if there is already a VPN client installed on your computer and otherwise install one automatically. You can find more detailed instructions on the ETH website.
Please note that for establishing a VPN connection, you need to use your network password instead of your main password. If you did not yet set your network password, then please go to https://password.ethz.ch, login with your ETH account credentials and click on Passwort ändern. There you can set your network password.
After establishing a VPN connection, you can login to our clusters via SSH.
Troubleshooting
Permission denied
If you enter 3 times a wrong password, then you will get a permission denied error:
sfux@calculus:~$ ssh sfux@euler.ethz.ch sfux@euler.ethz.ch's password: Permission denied, please try again. sfux@euler.ethz.ch's password: Permission denied, please try again. sfux@euler.ethz.ch's password: Permission denied (publickey,password,hostbased). sfux@calculus:~$
In case you receive a "Permission denied" error, please check if you entered the correct password. If you think that your account has been corrupted, then please contact the service desk of IT services of ETH Zurich.
If you enter a wrong password too many times or in a high frequency, then we might block access to the clusters for your account, because it could be correupted. If you account has been blocked by the HPC group, then please contact cluster support.
Timeout
If you try to login and receive a timeout error, then it is very likely that you tried to connect from outside of the ETH network to one of the HPC clusters.
sfux@calculus:~$ ssh -Y sfux@euler.ethz.ch ssh: connect to host euler.ethz.ch port 22: Connection timed out
Please either connect from the inside of the ETH network, or establish a VPN connection.
Setlocale: LC_CTYPE: cannot change locale (UTF-8): No such file or directory
If you are using a Mac, can you please try to comment out the following lines in your /etc/ssh/ssh_config on your workstation:
Host *
SendEnv LANG LC_*
This should solve the problem.
Too many authentication failures
This errors can be triggered if you have more than 6 private SSH keys in your local .ssh directory. In this case specify the SSH key to use and use the IdentitiesOnly=yes option, for example:
sfux@calculus:~$ ssh -i $HOME/.ssh/id_ed25519 -o IdentitiesOnly=yes sfux@euler.ethz.ch
Indirect GLX rendering error
When using an SSH connection with X11 forwarding enabled, newer versions of the Xorg server show an error message, when the graphical user interface of an application is started:
X Error of failed request: BadValue (integer parameter out of range for operation) Major opcode of failed request: 153 (GLX) Minor opcode of failed request: 3 (X_GLXCreateContext) Value in failed request: 0x0 Serial number of failed request: 27 Current serial number in output stream: 30
This error is caused by starting your X11 server without enabling the setting for indirect GLX rendering (iglx), that is required for X11 forwarding. Up to version 1.16 of the Xorg server, the setting iglx, has been enabled by default. With version 1.17, the default has changed from +iglx to -iglx. Now the setting needs to be enabled either in the Xorg configuration file or with a command line setting, when starting the Xorg server manually. For Xquartz versions up to 2.7.8, the iglx setting is enabled by default. If you would like to use XQuartz 2.7.9 or newer, then please make sure that you enable the iglx setting when the X-server is started.
This problem is described in the following article:
https://www.phoronix.com/scan.php?page=news_item&px=Xorg-IGLX-Potential-Bye-Bye
Please find below some links, which address the problem for specific operating systems.
| Operating system | Link |
|---|---|
| Red Hat Enterprise Linux (RHEL) | https://elrepo.org/bugs/view.php?id=610 |
| CentOS | https://www.centos.org/forums/viewtopic.php?t=57409#p244528 |
| Ubuntu | http://askubuntu.com/questions/745135/how-to-enable-indirect-glx-contexts-iglx-in-ubuntu-14-04-lts-with-nvidia-gfx |
| Mac OS X | https://bugs.freedesktop.org/show_bug.cgi?id=96260 |
