View source for Jupyter on Euler and Leonhard Open

<big style="color:red" >We are migrating towards a cloud based approach. Please migrate to [[JupyterHub]]</big>

==Introduction==
<big style="color:red" >We are migrating towards a cloud based approach. Please migrate to [[JupyterHub]]</big>

Jupyter notebooks and Jupyter Lab are becoming more widely used among the scientific community at ETH. We therefore developed a script that you can run on your local computer to start Jupyter notebooks or Jupyter lab on Euler. The shell script starts a Jupyter in a batch job on Euler and connects your local browser with it.

The script runs on Linux, macOS and Windows (using WSL/WSL2 or git bash).

Please note, that with this script we are addressing beginners that start to use Jupyter notebooks on the cluster. It is not addressing advanced users that need a wide range of additional features going beyond simple Jupyter notebooks. Advanced users can take the script and adapt it, such that it can be used with other Python versions (centrally installed, or local installations) and add support for GPU, adding new kernels etc.

==Installation==
Download the repository with the command
 git clone https://gitlab.ethz.ch/sfux/Jupyter-on-Euler-or-Leonhard-Open
Mac OS X:
 git clone https://gitlab.ethz.ch/sfux/Jupyter-on-Euler-or-Leonhard-Open.git
After downloading the script from gitlab.ethz.ch, you need to change its permissions to make it executable
cd Jupyter-on-Euler-or-Leonhard-Open/
chmod 755 start_jupyter_nb.sh

===Prerequisites===
In order to use this script, users need to make sure, that they have set up SSH keys for passwordless access to the cluster:

 https://scicomp.ethz.ch/wiki/Accessing_the_clusters#SSH_keys

Please make sure that xdg-open is installed. This package is used to automatically start your default browser. You can install it with the following command:

CentOS:

 yum install xdg-utils

Ubuntu:

 apt-get install xdg-utils

Further more, the script requires that there is a Python installation available, which is usually included in the Linux distribution or Mac OS.

===Security token vs. password setup===
Please note that a part of the script (parsing of the ports) requires that you use jupyter notebooks with the security tokens. If you configure a password instead, such that you can use the jupyter notebook without the security token, then the script will not work anymore (it cannot parse the port on the remote compute node) without adapting it.

===Download and setup===
The script is available on the Gitlab instance of ETH Zurich:

 https://gitlab.ethz.ch/sfux/Jupyter-on-Euler-or-Leonhard-Open

Download the repository with the command

 git clone https://gitlab.ethz.ch/sfux/Jupyter-on-Euler-or-Leonhard-Open

Mac OS X:

 git clone https://gitlab.ethz.ch/sfux/Jupyter-on-Euler-or-Leonhard-Open.git

After downloading the script from gitlab.ethz.ch, you need to change its permissions to make it executable

 chmod 755 start_jupyter_nb.sh

===Software stack===
Please note that currently the old software stack is still set a default (this will change). The script is using the new software stack (unless you explicitly request the old software stack with the option -s old (or --softwarestack old). Therefore please make sure that you set the new software stack as permanent default by using the command

 set_software_stack.sh new

You can find more information about this script on our wiki:

 https://scicomp.ethz.ch/wiki/Setting_permanent_default_for_software_stack_upon_login

===Creating SSH keys for the Jupyter script with a Windows computer (using git-bash)===
With git-bash, you can use the same command as for Linux/Mac to create your SSH keys:

* Start git-bash
* Run the command ssh-keygen to create the SSH key pair
 ssh-keygen -t ed25519 -f $HOME/.ssh/id_ed25519_euler
* Enter a passphrase to protect your SSH keys
* Use the ssh-copy-id command to copy your '''public''' (never your private) key to Euler
 ssh-copy-id -i ~/.ssh/id_ed25519_euler.pub '''username'''@euler.ethz.ch
* (here you need to replace '''username''' with your ETH username)
* Enter your password

After creating the keys, we strongly recommend to create an SSH config file in $HOME/.ssh

* Change to your .ssh directory
 cd $HOME/.ssh
* Create an SSH config file to specify the path to your SSH key
 echo -e "Host euler\nHostName euler.ethz.ch\nUser '''username'''\nIdentityFile ~/.ssh/id_ed25519_euler" > config 
* (here you need to replace '''username''' with your ETH username)

At this stage, passwordless login to Euler should work. You can test this with the command

 ssh username@euler.ethz.ch

* (here you need to replace '''username''' with your ETH username)

But now you still have to enter your passphrase. In order to not have to type your passphrase all the time, you can use an SSH agent, which is already included in git-bash

* Start git-bash
* Start the SSH agent for the current terminal
 ssh-agent /bin/bash
* Add your SSH key to the SSH agent
 ssh-add ~/.ssh/id_ed25519_euler
* Enter your passphrase

As long as this terminal is open, you can access Euler with
 ssh username@euler.ethz.ch
(here you need to replace '''username''' with your ETH username)
without having to type a password or passphrase. As soon as you exit the terminal, then the SSH agent will be stopped.

Now you are ready to clone the Jupyter script and to execute it in the terminal with the SSH agent. Please note that you only need to create the SSH once, but you need to setup the SSH agent every time you would like to run the Jupyter script.

===Using SSH keys with non-default names===
Since the reopening of Euler and Leonhard Open after the cyber attack in May 2020, we recommend to the cluster users to use SSH keys. We recommend to use different keys for Euler and Leonhard Open, with according names

 $HOME/.ssh/id_ed25519_euler

In order to use this keys with the jupyter script, then there are two options. You can configure your ssh client to use this option automatically by adding the following lines in the <tt>$HOME/.ssh/config</tt> file on your local computer:
 
 Host euler.ethz.ch
 IdentityFile ~/.ssh/id_ed25519_euler

Both options work and are equivalent.

===Installing additional Python packages locally===
When starting a Jupyter notebook with this script, then it will use a central Python installation:

* New software stack: module load gcc/6.3.0 python/3.8.5
* Old software stack: module load new gcc/4.8.2 python/3.6.1

Therefore you can only use packages that are centrally installed out-of-the-box. But you have the option to install additional packages locally in your home directory, which can afterwards be used.

For installing a Python package from inside a Jupyter notebook, you would need to run the following command:

 !pip install --user package_name

This will install <tt>package_name</tt> into <tt>$HOME/.local</tt>, as described on our wiki page about Python:

 https://scicomp.ethz.ch/wiki/Python#Installing_a_Python_package.2C_using_PIP

==Running the script==

The start_jupyer_nb.sh script needs to be executed on your local computer. Please find below the list of options that can be used with the script:

 $ ./start_jupyter_nb.sh: Script to start jupyter notebook/lab on Euler from a local computer
 
 Usage: start_jupyter_nb.sh [options]
 
 Required options:
 
         -u | --username       USERNAME         ETH username for SSH connection to Euler
 
 Optional arguments:
 
         -b | --batch_sys      BATCHSYS         Batch system to use (LSF or SLURM)
         -c | --config         CONFIG_FILE      Configuration file for specifying options
         -g | --numgpu         NUM_GPU          Number of GPUs to be used on the cluster
         -h | --help                            Display help for this script and quit
         -i | --interval       INTERVAL         Time interval for checking if the job on the cluster already started
         -j | --julia          BOOL             Start jupyter notebook with (BOOL=TRUE) or without (BOOL=FALSE) julia kernel enabled
         -k | --key            SSH_KEY_PATH     Path to SSH key with non-standard name
         -l | --lab                             Start jupyter lab instead of a jupyter notebook
         -m | --memory         MEM_PER_CORE     Memory limit in MB per core
         -n | --numcores       NUM_CPU          Number of CPU cores to be used on the cluster
         -s | --softwarestack  SOFTWARE_STACK   Software stack to be used (old, new)
         -v | --version                         Display version of the script and exit
         -w | --workdir        WORKING_DIR      Working directory for the jupyter notebook/lab
              --extra-modules  EXTRA_MODULES    Load additional cluster modules before starting
              --module-use     MODULE_USE       Use additional cluster module collection before starting
              --pythonpath     PYTHONPATH       Set PYTHONPATH before starting
         -W | --runtime        RUN_TIME         Run time limit for jupyter notebook/lab in hours and minutes HH:MM
 
 Examples:
 
         ./start_jupyter_nb.sh -u sfux -b SLURM -n 4 -W 04:00 -m 2048 -w /cluster/scratch/sfux
 
         ./start_jupyter_nb.sh -u sfux -b SLURM -n 1 -W 01:00 -m 1024 -j TRUE
 
         ./start_jupyter_nb.sh --username sfux --batch_sys SLURM --numcores 2 --runtime 01:30 --memory 2048 --softwarestack new
 
         ./start_jupyter_nb.sh -c /c/Users/sfux/.jnb_config
 
 Format of configuration file:
 
 JNB_USERNAME=""             # ETH username for SSH connection to Euler
 JNB_BATCH="SLURM"           # Choose SLURM or LSF as batch system
 JNB_EXTRA_MODULES           # Additional modules to be loaded
 JNB_NUM_CPU=1               # Number of CPU cores to be used on the cluster
 JNB_NUM_GPU=0               # Number of GPUs to be used on the cluster
 JNB_RUN_TIME="01:00"        # Run time limit for jupyter notebook/lab in hours and minutes HH:MM
 JNB_MEM_PER_CPU_CORE=1024   # Memory limit in MB per core
 JNB_WAITING_INTERVAL=60     # Time interval to check if the job on the cluster already started
 JNB_SSH_KEY_PATH=""         # Path to SSH key with non-standard name
 JNB_SOFTWARE_STACK="new"    # Software stack to be used (old, new)
 JNB_WORKING_DIR=""          # Working directory for jupyter notebook/lab
 JNB_ENV=""                  # Path to virtual environment
 JNB_JLAB=""                 # "lab" -> start jupyter lab; "" -> start jupyter notebook
 JNB_JKERNEL="FALSE"         # "FALSE" -> no Julia kernel; "TRUE" -> Julia kernel

[[File:jupyter_on_Euler.png|thumb|none|800px|Example for running a Jupyter notebook on the Euler cluster]]

===Reconnect to a Jupyter notebook===
When running the script, it creates a local file called reconnect_info in the installation directory, which contains all information regarding the used ports, the remote IP address, the command for the SSH tunnel and the URL for the browser. This information should be sufficient to reconnect to a Jupyter notebook if connection was lost.

===Terminate the Jupyter session===
Please note that when you finish working with the jupyter notebook, you need to click on the "Quit" or "Logout" button in your Browser. "Quit" will stop the batch job running on Euler, "Logout" will just log you out from the session but not stop the batch job (in this case you need to login to the cluster, identify the job with bjobs and then kill it with the bkill command, using the jobid as parameter). Afterwards you also need to clean up the SSH tunnel that is running in the background.

Example:

 samfux@bullvalene:~/Jupyter-on-Euler-or-Leonhard-Open$ ps -u | grep -m1 -- "-L" | grep -- "-N"
 samfux    8729  0.0  0.0  59404  6636 pts/5    S    13:46   0:00 ssh sfux@euler.ethz.ch -L 51339:10.205.4.122:8888 -N
 samfux@bullvalene:~/jupyter-on-Euler-or-Leonhard-Open$ kill 8729

==Troubleshooting==
===Logs===
If the script does not work, then we recommend to check the logs. There is on one hand the file <tt>$HOME/jnbinfo</tt> which contains logs, and on the other hand there is the lsf.o* logfile of the job that was running the jupyter notebook.

===Mismatch of software stacks===
In many cases it turns out that there are issues when the software stack used in the script does not match the default software stack that you have set. You can check which software stack is set as default by running the command

 set_software_stack.sh -i

The script has an option (-s/--softwarestack) for which you can specify either "old" or "new". Please either adapt the software stack that you are requesting for the script or change the default setting for the software stack that you use on the cluster.

===Conflict with locally installed Python packages===
When the jnbinfo contains a Python error with a backtrace, then it could be that some of the Python packages that you installed locally in <tt>$HOME/.local</tt> are conflicting with the centrally provided Jupyter installation.

==Jupyter in VSCode==
Another possibility to work with jupyter notebooks on Euler is to run them in VSCode (https://code.visualstudio.com/docs/datascience/jupyter-notebooks). You can run VSCode (actually code-server) on Euler using the script:

 https://scicomp.ethz.ch/wiki/VSCode