JupyterHub
Contents
- 1 Introduction
- 2 Warning
- 3 Prerequisites
- 4 Starting a session
- 5 Stopping your job
- 6 Debugging
- 7 Installing a Jupyterhub Extension
- 8 Disabling a Jupyterhub Extension
- 9 Other services
- 10 Known Issues
- 11 FAQ
- 11.1 I cannot login to the Jupyter service
- 11.2 My server is too slow to start
- 11.3 My server has been killed before starting
- 11.4 My server has been killed even if my job had still plenty of remaining time
- 11.5 My Jupyter kernel died and I cannot restart a new one
- 11.6 I cannot request resources for more than 24h
- 11.7 Jupyter Build Recommended but fails
- 11.8 My job with N GPUs is not starting
- 11.9 I lost all my Jupyter settings when migrating from the script to the hub
- 11.10 I want to load a cluster module / I want to activate a virtualenv / Jupyterlab is missing some features
- 11.11 I wish to use custom arguments to jupyterhub-singleuser
- 11.12 I can't install a Julia package
Introduction
JupyterHub on Euler is used to provide many cloud services and not only Jupyter. Among them, you can find Tensorboard, VS Code (also called "Code Server", "Visual Code", "Visual Studio Code" or "VSCode").
JupyterLab and Jupyter notebooks are widely used in the scientific community at ETH as they provide an easy way to run Python code (or to use other programming languages) in a browser window. We therefore developed a service that allows users to start a JupyterLab session in their browser without having to login to Euler via an SSH client. It provides an easy access to computational resources of the Euler cluster and you can use it to interactively work and to develop and test your code. You can access it here.
Warning
This service is made for developing and debugging your code, not to run your code on the cluster. If you wish to run it, please use the queue system.
GPUs nodes (especially high end ones) tend to have a high IO / network load so the starting of a session can be extremely slow and can even be killed by Jupyterhub due to it. We are aware of this and we are not planning to improve it due to the previous point. If you need a GPU and cannot get a session, please use a script and submit it to the queue.
Prerequisites
The only prerequisite to use this service is that you have a local computer with a browser installed. As the Euler cluster itself, the service can only be used from within the ETH network. If you are working from home, then you would first need to establish a VPN connection to the ETH network.
Please note that if you have never logged into the Euler cluster before using this service, then you first need to login once with an SSH client to verify your ETH account and to accept the clusters usage rules. Please check our wiki page about accessing the cluster. On this page you can find all information required to login to the Euler cluster with your SSH agent. When you login for the first time, an access code will be sent to your ETH email address that you need to enter and then you need to accept the clusters usage rules. After this initial procedure you can use the Jupyter service.
Starting a session
You can start a session by opening your favorite browser and by entering the following URL https://jupyter.euler.hpc.ethz.ch. Then you will be asked to login with your ETH credentials. After entering your ETH credentials and clicking on the Sign in button, you can choose the amount of resources that you request for your session. Please only request multiple cores if you are planning to run some code that can make use of multiple cores. By clicking on the Start button, a batch job with your session will be started. It might takes some time until the batch job has started, but then JupyterLab will start in your browser window.
Please note that the service is using the module system on Euler. It has several hundred packages preinstalled that you can use right away in your session when starting a Python kernel.
You can find a comprehensive tutorial about JupyterLab on
https://jupyterlab.readthedocs.io/en/stable/getting_started/overview.html
Stopping your job
Please don't forget to kill your job when you are done with your JupyterLab (or stop your server in jupyterhub)
If you just stop the current kernel or close the browser window, then the batch job on Euler will continue to run and waste resources that could be used by other cluster users. To properly stop a jupyter session, you need to click the File menu and choose the entry Hub Control Panel (see picture) and then click the Stop my server button. Afterwards you can close the browser window and your session is terminated.
If you don't have access to this menu (e.g. in tensorboard or other services), you can also access the hub by changing the URL. You just need to replace everything after /user (included) by /hub/home.
Debugging
Before opening a ticket, please check the logs of your jupyterlab. They are available in the following directory $HOME/jupyterhub-logs/. If you are not able to debug it yourself, please add this file to the ticket.
Installing a Jupyterhub Extension
It is possible to extend the basic functionality of JupyterLab with extensions. We provide some preinstalled extensions for the users, but there are probably still some useful extensions missing. You can not directly use the extension manager from JupyterLab as this would required write permission in the central installation directory of JupyterLab which users don't have. There is no easy way to configure JupyterLab to store the extensions in a user-writable directory. For some extensions it is possible to install them with pip:
For example if you wish to install jupyterlab-slurm, you will need to run the following commands:
module load REQUIRED_MODULES pip3 install --user jupyterlab_slurm jupyter labextension enable jupyterlab_slurm
where REQUIRED_MODULES are the ones required by Jupyterlab. In order to have the current configuration, please look at the top of your log files ($HOME/jupyterhub-logs).
If an extension for JupyterLab is useful for many users, then you can also ask cluster-support if the extension can be installed centrally.
Disabling a Jupyterhub Extension
If you are unhappy with an extension, you can disable it with:
jupyter labextension disable my-extension
Other services
By using a proxy on the server, we can provide other services within jupyterhub.
Feel free to copy the settings of Jupyter (in its logs) / tensorboard / VS Code into ~/.config/euler/jupyterhub/jupyterlabrc to create your own web services.
Tensorboard
Tensorboard can be selected when starting the server in the option Software. Unfortunately, Tensorboard does not provide any authentication method, therefore anyone can access your tensorboard if they know where you run it and on which port.
Code-server
Code-server can be selected when starting the server in the option Software. If you need to load some modules before starting VS code, you can create the file ~/.config/euler/jupyterhub/config_vs_code (bash script).
If you wish to see how we start a code-server session, the code is available in /cluster/software/others/services/jupyterhub/scripts/code-server.sh.
RStudio
RStudio can be selected when starting the server in the option Software. To start a RStudio session, you will need to have access to Singularity. If you need to load some modules before starting RStudio, you can create the file ~/.config/euler/jupyterhub/config_r_studio (bash script).
Credentials
The credentials are provided in ~/.rstudio/.password.
Installing a package
You will need to setup the proxy before installing anything:
Sys.setenv(https_proxy=http://proxy.ethz.ch:3128) Sys.setenv(http_proxy=http://proxy.ethz.ch:3128)
Known Issues
- Currently, the plugins cannot be installed directly from the UI. Please use the command line to install them
FAQ
I cannot login to the Jupyter service
If it is the first time that you are using Euler, you will need to connect first with SSH. Please read this page for more information on how to do it.
My server is too slow to start
We rely on the Slurm batch system to provide the JupyterLab instances. So it could be either due to a low amount of available resources in Euler or that your priority is too low (already used too much resources or too many jobs running at the same time).
My server has been killed before starting
JupyterHub relies on a timeout system to manage the starting jobs (currently around 10 minutes). If your job takes more time than that to start, it will be automatically killed. If you are unable to get one after multiple tries, please check your queue by using ssh and running squeue on Euler.
My server has been killed even if my job had still plenty of remaining time
Jupyterhub regularly checks jobs for activity. If you do not use your notebook / lab for too long, jupyterhub will kill your notebook. We will not change this behavior as it improves the fair usage of the cluster for everyone.
My Jupyter kernel died and I cannot restart a new one
This could be due to a lack of memory. Please check your logs ($HOME/jupyterhub-logs) to see if you have a OOM message from slurm. You can also check the memory usage with myjobs -j JOB_ID. If you reach something above 90-95%, it could indicate a memory issue. To fix this, you just need to request more RAM when starting a jupyterlab.
I cannot request resources for more than 24h
This service aims at cluster beginners and therefore we chose to only allow short sessions up to 24 hours. For running longer jobs for more than 24h, we recommend to submit them directly to the queue and to not use JupyterLab for that.
Jupyter Build Recommended but fails
JupyterLab is trying to build all its files within the system directories which is of course not allowed. No worries about this issue, we will try to keep up to date the JupyterLab, but we will not do it with every minor releases of a plugin.
My job with N GPUs is not starting
GPUs are only available to shareholders that purchased GPU resources in Euler. Please ensure that you indeed have access to GPUs on Euler before submitting a ticket to cluster support.
I lost all my Jupyter settings when migrating from the script to the hub
With the JupyterHub, we are using the directory ~/.jupyterlab and not ~/.jupyter to store all the configurations. Replacing the content of the new directory by the old one should be sufficient.
I want to load a cluster module / I want to activate a virtualenv / Jupyterlab is missing some features
You can add your own instruction by writing your own bash script in ~/.config/euler/jupyterhub/jupyterlabrc. If you don't use Jupyter, some equivalent files are provided. See the other services
This script will be sourced (. ~/.config/euler/jupyterhub/jupyterlabrc) before starting any service. So you can load some modules, update some environment variables, replace jupyterlab by another service (advanced usage: see how tensorboard is done), ...
For using a virtual environment, you need to register it as ipykernel with the command
python -m ipykernel install --user --name=myenv
Then the environment will be able as python kernel myenv in jupyter.
I wish to use custom arguments to jupyterhub-singleuser
A few environment variables can be defined in your ~/.config/euler/jupyterhub/jupyterlabrc file:
- JUPYTER_DIR: Available directory for the users - JUPYTER_HOME: Default directory - JUPYTER_EXTRA_ARGS: any additional argument (e.g. '--debug')
I can't install a Julia package
If you get a permission denied when installing a Julia package, you will need first to activate a local directory:
using Pkg Pkg.activate(".") Pkg.add("PACKAGE_NAME")