Difference between revisions of "Accessing the cluster"

From ScientificComputing
Jump to: navigation, search
(62 intermediate revisions by the same user not shown)
Line 1: Line 1:
 
__NOTOC__
 
__NOTOC__
 +
<table style="width: 100%;">
 +
<tr valign=top>
 +
<td style="width: 50%; text-align:left">
 +
< [[Main_Page|Home]]
 +
</td>
 +
<td style="width: 50%; text-align:right">
 +
[[Storage and data transfer]] >
 +
</td>
 +
</tr>
 +
</table>
 +
 +
 
<table style="width: 100%;">
 
<table style="width: 100%;">
 
<tr valign=top>
 
<tr valign=top>
 
<td style="width: 45%;">
 
<td style="width: 45%;">
−
== Who can access the cluster ==
+
[[File:Accessing the clusters.png|600px]]
−
{| class="wikitable" | style="background: white;"
 
−
! || Euler || Leonhard Open
 
−
|-
 
−
| Shareholders who invested into a share of the clusters resources  ||style="color:green;"| Yes ||style="color:green;"| Yes
 
−
|-
 
−
| External collaboration partners of shareholders ||style="color:green;"| Yes ||style="color:green;"| Yes
 
−
|-
 
−
| All ETH members ||style="color:green;"| Yes, as guest users with limited resources ||style="color:red;"| No
 
−
|}
 
 
</td>
 
</td>
 
<td style="width: 5%;>
 
<td style="width: 5%;>
 
</td>
 
</td>
−
<td style="width: 45%;">
+
<td style="width: 50%;">
  
 
== Prerequisites ==
 
== Prerequisites ==
 
* A valid ETH account
 
* A valid ETH account
−
* [[Accessing_the_clusters#SSH|Local computer with an SSH client]]
+
* Local computer with an SSH client
−
* [[Accessing_the_clusters#X11| An X11 server for graphical user interface]] (optional)
+
** Linux and macOS contain SSH client in the operating system
 +
** Windows users need to install a third party SSH client, e.g., [https://mobaxterm.mobatek.net/ MobaXterm], [https://www.putty.org/ PuTTY], [https://gitforwindows.org/ Git BASH] or [https://docs.microsoft.com/en-us/windows/wsl/install Windows Subsystem Linux (WSL)]
 +
* [[Accessing_the_clusters#X11| An X11 server for graphical user interface (GUI)]] (optional)
 +
** Linux: [https://www.xorg.com Xorg] is usually installed
 +
** macOS users need to install [https://xquartz.org XQuartz]
 +
** Windows: MobaXterm includes and enables an X11 server
 
</td>
 
</td>
 
</tr>
 
</tr>
 +
</table>
  
−
 
+
== How to access the cluster ==
−
 
+
<table style="width: 100%;">
 
<tr valign=top>
 
<tr valign=top>
 
<td style="width: 45%;">
 
<td style="width: 45%;">
−
== How to access the cluster ==
+
* Start your SSH client  
−
* Start your SSH client
+
* From a terminal (Linux, MacOS, Git BASH, WSL), use ssh command to connect to the login node of Euler
−
* Use ssh command to connect to the login node of Euler or Leonhard Open
+
   $ ssh username@euler.ethz.ch
−
   ssh username@euler.ethz.ch
+
* Use your ETH credentials to log in
−
  ssh username@login.leonhard.ethz.ch
+
* See here for [[How to access the cluster with PuTTY|how to access the cluster with PuTTY]]
−
* Use your ETH credentials to login
 
−
* [[Accessing_the_clusters#SSH_keys|Generate SSH keys for passwordless login]]
 
−
 
 
 
</td>
 
</td>
 
<td style="width: 5%;>
 
<td style="width: 5%;>
 
</td>
 
</td>
 
<td style="width: 45%;">
 
<td style="width: 45%;">
−
== For new users and unverified accounts ==
+
 
 +
 
 +
</td>
 +
</tr>
 +
</table>
 +
 
 +
<table style="width: 100%;">
 +
<tr valign=top>
 +
<td style="width: 45%;">
 +
 
 +
=== For new users and unverified accounts ===
 
* Upon the first login, a verification code should be sent to your ETH email address (username@ethz.ch)
 
* Upon the first login, a verification code should be sent to your ETH email address (username@ethz.ch)
 
* Enter the verification code to the prompt
 
* Enter the verification code to the prompt
Line 54: Line 70:
  
 
* [[Accessing_the_clusters#First_login|Accept the cluster’s usage rules upon the first access]]
 
* [[Accessing_the_clusters#First_login|Accept the cluster’s usage rules upon the first access]]
 +
 +
</td>
 +
<td style="width: 5%;>
 +
</td>
 +
<td style="width: 45%;">
 +
 +
=== Login options ===
 +
* To use GUI on the cluster
 +
**For Windows users, X11 is already included and enabled in MobaXterm.
 +
**Linux and macOS users have to enable X11 forwarding when log in
 +
    $ ssh -Y username@euler.ethz.ch
 +
 +
* After a successful login with password, [[Accessing_the_clusters#SSH_keys|generate SSH keys for passwordless login]]
 +
  
 
</td>
 
</td>
Line 59: Line 89:
 
</table>
 
</table>
  
−
== External collaborators ==
+
== Who can access the cluster ==
 +
<table style="width: 100%;">
 +
<tr valign=top>
 +
<td style="width: 45%;">
 +
 
 +
{| class="wikitable" | style="background: white;"
 +
! || Euler CPU || Euler GPU
 +
|-
 +
| Shareholders who invest in the cluster resources || style="color:green; text-align: center;"| &#x2713; <br /> Euler CPU shareholders ||style="color:green;text-align: center;"| &#x2713; <br /> Euler GPU shareholders
 +
|-
 +
| External collaborators of shareholders || style="color:green;text-align:center;"| &#x2713; ||style="color:green;text-align: center;"| &#x2713;
 +
|-
 +
| Guest users || style="color:green;text-align:center;"| &#x2713; <br />All ETH members can access Euler as guest users with limited resources||style="color:red;text-align:center;"| &#x2715;
 +
|}
 +
 
 +
</td>
 +
<td style="width: 5%;>
 +
</td>
 +
<td style="width: 45%;">
 +
=== External collaborators ===
 
Members of other institutions who have a collaboration with a research group at ETH may use the clusters for the purpose of said collaboration
 
Members of other institutions who have a collaboration with a research group at ETH may use the clusters for the purpose of said collaboration
 
* Their counterpart (“sponsor”) at ETH can create an ETH guest account, e-mail address and VPN service for them
 
* Their counterpart (“sponsor”) at ETH can create an ETH guest account, e-mail address and VPN service for them
 
* Then, they can access Euler like members of ETH
 
* Then, they can access Euler like members of ETH
 +
</td>
 +
</tr>
 +
</table>
  
−
== Firewall ==
+
== Security ==
−
* To connect from outside of the ETH network to the clusters, establish first a [[Accessing_the_clusters#VPN|VPN connection]]. Then, connect to the cluster through SSH.
+
<table style="width:100%">
−
* To connect to an external service from a compute node, use the ETH proxy service:
+
<tr valign=top>
−
  module load eth_proxy
+
<td style="width: 40%">
−
[[Image:cluster.png|660px|]]
+
* To connect from outside of the ETH network to the cluster, establish first a [[Accessing_the_clusters#VPN|VPN connection]]. Then, connect to the cluster through SSH.
 +
* To connect from a compute node to an external service, use the ETH proxy service:
 +
  $ module load eth_proxy
  
−
== Legal Compliance ==
+
=== Legal Compliance ===
 
The HPC clusters are subject to ETH’s acceptable use policy for IT resources ([https://rechtssammlung.sp.ethz.ch/Dokumente/203.21en.pdf Benutzungsordnung für Telematik (BOT)]), in particular:
 
The HPC clusters are subject to ETH’s acceptable use policy for IT resources ([https://rechtssammlung.sp.ethz.ch/Dokumente/203.21en.pdf Benutzungsordnung für Telematik (BOT)]), in particular:
 
* Cluster accounts are strictly personal
 
* Cluster accounts are strictly personal
−
* '''DO NOT''' share your account (password, sshkeys) with anyone
+
* DO NOT share your account (password, sshkeys) with anyone
−
* '''DO NOT''' use someone else’s account, even if they say it’s OK
+
* DO NOT use someone else’s account, even if they say it’s OK
−
* If you suspect that someone used your account, [https://www.password.ethz.ch change your password] and contact cluster-support@id.ethz.ch
+
* If you suspect that someone used your account, [https://www.password.ethz.ch change your password] and contact [mailto:cluster-support@id.ethz.ch cluster support]
−
 
+
</td>
 +
<td style="width: 5%">
 +
</td>
 +
<td style="width: 55%">
 +
[[Image:security.png|600px]]
 +
</td>
 +
</tr>
 +
</table>
 
Consequences:
 
Consequences:
  
 
* In case of abuse, the offender’s account may be blocked temporarily or closed
 
* In case of abuse, the offender’s account may be blocked temporarily or closed
 
* System administrators are obliged by law to investigate abusive or illegal activities and report them to the relevant authorities
 
* System administrators are obliged by law to investigate abusive or illegal activities and report them to the relevant authorities
 +
 +
== SSH Keys ==
 +
* SSH keys allows passwordless login which is useful for file transfers and automated tasks
 +
* When used properly, SSH keys are much safter than passwords
 +
* SSH keys always come in pairs:
 +
** '''A private key''', stored on your local workstation (and nowhere else!)
 +
** '''A public key''', stored on the computer(s) you want to connect to
 +
** You can generate as many pairs as you like, e.g., one for each computer you intend to connect to
 +
* Keys should be protected with a passphrase
 +
* SSH key management
 +
** [https://www.ssh.com/academy/ssh/agent#starting-ssh-agent ssh-agent] unlocks keys for each terminal
 +
** [https://linux.die.net/man/1/keychain keychain] activates ssh-agent in every new terminal
 +
 +
<table>
 +
<tr valign=center>
 +
<td style="width: 30%; background: white;">
 +
=== Step 1: Create your keys ===
 +
</td>
 +
</tr>
 +
<tr valign=center>
 +
<td style="width: 30%; background: white;">
 +
<br>
 +
[[Image:ssh_keys_gen.png|370px]]
 +
<br>
 +
</td>
 +
<td style="width: 60%; background: white; text-align: left">
 +
* First, verify whether logging in with password works
 +
* Generate a key pair with the ed25519 algorithm for each computer you want to connect to
 +
 +
$ ssh-keygen -t ed25519 -f $HOME/.ssh/id_ed25519_euler
 +
 +
* Enter a passphrase to protect your SSH keys
 +
</td>
 +
</tr>
 +
 +
<tr valign=center>
 +
<td style="width: 30%; background: white;">
 +
=== Step 2: Copy the public key to the cluster ===
 +
</td>
 +
</tr>
 +
 +
<tr valign=center>
 +
<td style="width: 30%; background: white;">
 +
[[Image:ssh_keys_copy.png|370px]]
 +
</td>
 +
<td style="width: 60%; background: white; text-align: left;">
 +
$ ssh-copy-id -i $HOME/.ssh/id_ed25519_euler.pub username@euler.ethz.ch
 +
</td>
 +
</tr>
 +
 +
<tr valign=center>
 +
<td style="width: 30%; background: white;">
 +
=== Step 3: Use keys with non-default names ===
 +
</td>
 +
</tr>
 +
 +
<tr valign=top>
 +
<td style="width: 30%; background: white;">
 +
<br>
 +
[[Image:ssh_keys_connect.png|370px]]
 +
<br>
 +
</td>
 +
<td style="width: 60%; background: white; text-align: left;">
 +
The login commands become:
 +
 +
$ ssh -i $HOME/.ssh/id_ed25519_euler username@euler.ethz.ch
 +
 +
Alternatively, SSH clients can use this option automatically by adding the option IdentityFile in your $HOME/.ssh/config file, e.g.:
 +
 +
Host euler
 +
HostName euler.ethz.ch
 +
User username
 +
IdentityFile ~/.ssh/id_ed25519_euler
 +
 +
Next time you login, you can type
 +
 +
$ ssh euler
 +
 +
</td>
 +
</tr>
 +
</table>
 +
 +
=== SSH key management with SSH Agent ===
 +
As we have to enter the passphrase to unlock the keys, it takes away the convenience of passwordless login. We can use an SSH agent (ssh-agent) to unlock the SSH keys.
 +
$ eval `ssh-agent`
 +
Agent pid 17906
 +
 +
$ ssh-add $HOME/.ssh/id_ed25519_euler
 +
Enter passphrase for id_ed25519_euler:
 +
Identity added: id_ed15519_euler (username@local-computer-name)
  
 
== Further reading ==
 
== Further reading ==
 
* [[New_account_request_process_for_HPC_clusters| New account request process for HPC clusters]]
 
* [[New_account_request_process_for_HPC_clusters| New account request process for HPC clusters]]
 
* [[Accessing_the_clusters|User guide: Accessing the clusters]]
 
* [[Accessing_the_clusters|User guide: Accessing the clusters]]
 +
* [[Getting_started_with_clusters#Troubleshooting | Troubleshooting]]
 +
 +
 +
<table style="width: 100%;">
 +
<tr valign=top>
 +
<td style="width: 50%; text-align:left">
 +
< [[Main_Page|Home]]
 +
</td>
 +
<td style="width: 50%; text-align:right">
 +
[[Storage and data transfer]] >
 +
</td>
 +
</tr>
 +
</table>

Revision as of 15:24, 22 December 2021

< Home

Storage and data transfer >


Accessing the clusters.png

Prerequisites

How to access the cluster

  • Start your SSH client
  • From a terminal (Linux, MacOS, Git BASH, WSL), use ssh command to connect to the login node of Euler
 $ ssh username@euler.ethz.ch


For new users and unverified accounts

  • Upon the first login, a verification code should be sent to your ETH email address (username@ethz.ch)
  • Enter the verification code to the prompt
An access code has been sent to your ETH email address.
Enter the access code at the prompt below.
If you do not receive the access code within a few minutes,
then contact us by opening a ticket at
http://smartdesk.ethz.ch

Access code (ending in ******Ls):

Login options

  • To use GUI on the cluster
    • For Windows users, X11 is already included and enabled in MobaXterm.
    • Linux and macOS users have to enable X11 forwarding when log in
   $ ssh -Y username@euler.ethz.ch


Who can access the cluster

Euler CPU Euler GPU
Shareholders who invest in the cluster resources ✓
Euler CPU shareholders 		✓
Euler GPU shareholders
External collaborators of shareholders ✓ ✓
Guest users ✓
All ETH members can access Euler as guest users with limited resources		 ✕

External collaborators

Members of other institutions who have a collaboration with a research group at ETH may use the clusters for the purpose of said collaboration

  • Their counterpart (“sponsor”) at ETH can create an ETH guest account, e-mail address and VPN service for them
  • Then, they can access Euler like members of ETH

Security

  • To connect from outside of the ETH network to the cluster, establish first a VPN connection. Then, connect to the cluster through SSH.
  • To connect from a compute node to an external service, use the ETH proxy service:
$ module load eth_proxy

Legal Compliance

The HPC clusters are subject to ETH’s acceptable use policy for IT resources (Benutzungsordnung für Telematik (BOT)), in particular:

  • Cluster accounts are strictly personal
  • DO NOT share your account (password, sshkeys) with anyone
  • DO NOT use someone else’s account, even if they say it’s OK
  • If you suspect that someone used your account, change your password and contact cluster support

Security.png

Consequences:

  • In case of abuse, the offender’s account may be blocked temporarily or closed
  • System administrators are obliged by law to investigate abusive or illegal activities and report them to the relevant authorities

SSH Keys

  • SSH keys allows passwordless login which is useful for file transfers and automated tasks
  • When used properly, SSH keys are much safter than passwords
  • SSH keys always come in pairs:
    • A private key, stored on your local workstation (and nowhere else!)
    • A public key, stored on the computer(s) you want to connect to
    • You can generate as many pairs as you like, e.g., one for each computer you intend to connect to
  • Keys should be protected with a passphrase
  • SSH key management
    • ssh-agent unlocks keys for each terminal
    • keychain activates ssh-agent in every new terminal

Step 1: Create your keys


Ssh keys gen.png

  • First, verify whether logging in with password works
  • Generate a key pair with the ed25519 algorithm for each computer you want to connect to
$ ssh-keygen -t ed25519 -f $HOME/.ssh/id_ed25519_euler
  • Enter a passphrase to protect your SSH keys

Step 2: Copy the public key to the cluster

Ssh keys copy.png 

$ ssh-copy-id -i $HOME/.ssh/id_ed25519_euler.pub username@euler.ethz.ch

Step 3: Use keys with non-default names


Ssh keys connect.png

The login commands become: 

$ ssh -i $HOME/.ssh/id_ed25519_euler username@euler.ethz.ch

Alternatively, SSH clients can use this option automatically by adding the option IdentityFile in your $HOME/.ssh/config file, e.g.: 

Host euler
HostName euler.ethz.ch
User username
IdentityFile ~/.ssh/id_ed25519_euler

Next time you login, you can type 

$ ssh euler

SSH key management with SSH Agent

As we have to enter the passphrase to unlock the keys, it takes away the convenience of passwordless login. We can use an SSH agent (ssh-agent) to unlock the SSH keys. 

$ eval `ssh-agent`
Agent pid 17906

$ ssh-add $HOME/.ssh/id_ed25519_euler
Enter passphrase for id_ed25519_euler:
Identity added: id_ed15519_euler (username@local-computer-name)

Further reading


< Home

Storage and data transfer >

Retrieved from "https://scicomp.ethz.ch/w/index.php?title=Accessing_the_cluster&oldid=6652"