Migrating to Ubuntu

From ScientificComputing
Jump to: navigation, search

Introduction

The operating system of Euler, CentOS 7, will reach its end-of-life on 30 June 2024. No new version of CentOS will be released in the future, and no security updates will be published for the current version. The only way to provide a secure computing environment after this date is to migrate the whole Euler cluster to a more recent operating system (OS).

After considering different options (Rocky Linux, RHEL, Ubuntu) we decided to migrate to Ubuntu 22.04 LTS. (The upcoming 24.04 LTS cannot be used because it is not yet compatible with some of the hardware in Euler.) Our goal is to make this transition as smooth as possible to the users, but since we are dealing with a different Linux distribution, some changes and adjustments (e.g. regarding workflows) are unavoidable.

What is new

Ubuntu 22.04 LTS provides a much newer kernel (5.15) than CentOS 7 (3.10), and well as newer versions of Glibc. This will simplify the installation of newer software, and should also bring some performance benefits, due to better support for newer CPU architectures.

Some applications, scripts and workflows may need to be recompiled and/or adapted to run on Ubuntu. We are launching today a beta testing phase to let you check if your codes run correctly on Ubuntu, and to make some adjustments if necessary.

What stays the same

The migration to the new OS does not affect your data. All files in /cluster/home, /cluster/scratch, /cluster/project and /cluster/work will remain exactly the same.

The Slurm batch system has already been ported to Ubuntu and is working fine. There will be no changes to the shareholder model or to the Slurm partitions, except during the beta testing phase, where some functionalities will be limited.

Timeline

The migration will be done in stages as follows:

  • Early April: beta phase 1, Ubuntu installed on some login nodes and CPU nodes
  • Early May: beta phase 2, Ubuntu also installed on some GPU nodes
  • Late May: official release of Ubuntu on Euler, operating side-by-side with CentOS
  • 4-6 June: Migration of Jupyterhub to Ubuntu during the maintenance
  • June: progressive migration of compute nodes from CentOS to Ubuntu
  • 30 June: retirement of CentOS

In parallel, we will introduce a new software stack compiled specially for the new OS, starting with the most commonly used software. Due to complex support issues, the installation of some commercial software may take some time.

Information for beta users

Login

The login procedure is the same as on CentOS but with a different hostname:

ssh username@login-beta.euler.ethz.ch

Software stack

We are providing a new software stack for the new OS, which is set up a bit differently from the previous one. This is still in progress. While we work on it, we already provide a beta release which will be very similar to the production stack. The beta software stack will be replaced with a new production one once the beta test phase is over.

In order to improve the isolation between stacks and commercial software, the stacks will require a module load. You can see available versions and load the most recent one with

module avail stack
module load stack

Please not that many low-level modules (i.e., modules needed only by other modules) are now hidden by default, so module avail and module spider will only show you the higher-level modules. To see all modules, you can use the commands

module --show_hidden avail
module --show_hidden spider SOFTWARE

We are also providing a way for users to install their own software stack on top of the centrally-provided one using spack.

Work in progress

Here you can find a list of software that we are about to install in the new software environment.

  • Abaqus 2023
  • Ansys 2023.1
  • CVS
  • Gurobi
  • Gams
  • Hyperworks 2021.2

User environment

You may need to edit your ~/.bashrc to work with the new OS, in particular if you are loading some modules by default, since the module names/versions may differ in the new OS.

Slurm

A separate instance of Slurm is deployed on all beta nodes (login and compute), so you cannot submit jobs to Ubuntu nodes from CentOS nodes or the other way around.

The number of beta compute nodes is currently very limited, which makes it hard to run MPI jobs across multiple nodes. Beta users should therefore focus on single-processor and shared-memory (openMP) parallel jobs.

The maximum run-time is 24h.

Jupyterhub

The service will be migrated between the 4th-6th June. The configuration files are now read from ~/.config/euler/jupyterhub. Please move them manually if you wish to conserve your configuration.


Backward compatibility

In order to simplify the migration, we provide a compatibility tool with the command run-centos7 which can be used to get a shell (e.g. run-centos7) or to run a command (e.g. run-centos7 ls) in a CentOS 7 container running on top of Ubuntu.

We strongly recommend to do a full migration and not rely on this script on the long term as we will be slowly dropping the support on anything related to CentOS (e.g. software stack).

Support

In case of problem, please check the "Known issues" below. If your problem is not listed, please contact cluster-support@id.ethz.ch and mention that your issue is with the Ubuntu beta.

Known issues

  • For certain software, the beta stack contains the same version twice (for instance llvm, openmm). We will fix this for the new stack that is deployed after the beta test is finished.
  • The performances of openmpi are low. This will be fixed in the production software stack. If you wish to test it, you can set MODULEPATH=/cluster/software/stacks/2024-05/spack/share/spack/lmod/linux-ubuntu22.04-x86_64/Core:/cluster/software/lmods and load the required modules (we don't provide any guarantee on the availability of this stack as we are working on it)