Difference between revisions of "Integration of Leonhard Open into Euler"

From ScientificComputing
Jump to: navigation, search
(Introduction)
Line 1: Line 1:
 
==Introduction==
 
==Introduction==
The Leonhard Cluster was introduced in 2017 as a new platform for big data analytics featuring petabytes of high-performance storage and GPU nodes for machine learning applications to process large amounts of data. At this time, the Euler cluster did not have any GPU nodes available and could not cover this new use-cases that caught the attention of more and more researchers. Over the years, we also introduced GPU nodes on Euler and increased the capacity of the high-performance storage, such that both systems have become very similar in terms of use cases that they can cover. From an operational point of view it makes a lot of sense to integrate the Leonhard Open cluster into Euler as this allows us to exploit synergies in the system management, software installation and support.  
+
The Leonhard Open Cluster was introduced in 2017 as a new platform for big data analytics featuring petabytes of high-performance storage and GPU nodes for machine learning applications to process large amounts of data. At this time, the Euler cluster did not have any GPU nodes available and could not cover this new use-cases that caught the attention of more and more researchers. Over the years, we also introduced GPU nodes on Euler and increased the capacity of the high-performance storage, such that both systems have become very similar in terms of use cases that they can cover. From an operational point of view it makes a lot of sense to integrate the Leonhard Open cluster into Euler as this allows us to exploit synergies in the system management, software installation and support.  
  
The Leonhard Open shares will be transferred to Euler. This migration happens in a transparent way and apart from the hostname (euler.ethz.ch instead of login.leonhard.ethz.ch) nothing will change for the users. Both software stacks feature the same toolchains (GCC 4.8.5, 6.3.0, 8.2.0 and Intel 18.0.1) and the same batch system. We already started to install packages on Euler that were before only available on Leonhard Open to make the migration of the workflows as seamless as possible. Still there will be some installations missing and we will be happy to assist you in porting your workflows from Leonhard Open to Euler.
+
The Leonhard Open shares will be transferred to Euler. This migration happens in a transparent way and apart from the hostname (euler.ethz.ch instead of login.leonhard.ethz.ch) nothing will change for the users. Both software stacks feature the same toolchains (GCC 4.8.5, 6.3.0, 8.2.0 and Intel 18.0.1) and both clusters are running the same batch system. We already started to install packages on Euler that were before only available on Leonhard Open to make the migration of the workflows as seamless as possible. Still there will be some installations missing and we will be happy to assist you in porting your workflows from Leonhard Open to Euler.
  
 
==Data transfer==
 
==Data transfer==
Line 12: Line 12:
 
==FAQ==
 
==FAQ==
 
===What happens to my Leonhard Open share?===
 
===What happens to my Leonhard Open share?===
Nothing. You will get exactly the same resources that you had on Leonhard Open for the remaining part of the 4 years duration that a share is valid.
+
Nothing. You will get exactly the same resources that you had on Leonhard Open on Euler for the remaining part of the 4 years duration that a share is valid. Since the Euler cluster is much larger than the Leonhard Open cluster, there will be a larger elasticity allowing for higher peak usage.
  
 
===Do I need to transfer my data to Euler?===
 
===Do I need to transfer my data to Euler?===
 
You will need to transfer data from your home directory and your personal scratch directory. We will take care of transferring the data from your work storage shares.
 
You will need to transfer data from your home directory and your personal scratch directory. We will take care of transferring the data from your work storage shares.
 +
 
===Is all the software that was available on Leonhard Open also available on Euler?===
 
===Is all the software that was available on Leonhard Open also available on Euler?===
 
To prepare the integration of Leonhard Open into Euler we already started some months ago to identify differences in the software stacks of Leonhard Open and Euler and to install missing software packages on Euler that were only available on Leonhard Open. There will still be some packages missing that we can install again on request.
 
To prepare the integration of Leonhard Open into Euler we already started some months ago to identify differences in the software stacks of Leonhard Open and Euler and to install missing software packages on Euler that were only available on Leonhard Open. There will still be some packages missing that we can install again on request.

Revision as of 06:09, 19 August 2021

Introduction

The Leonhard Open Cluster was introduced in 2017 as a new platform for big data analytics featuring petabytes of high-performance storage and GPU nodes for machine learning applications to process large amounts of data. At this time, the Euler cluster did not have any GPU nodes available and could not cover this new use-cases that caught the attention of more and more researchers. Over the years, we also introduced GPU nodes on Euler and increased the capacity of the high-performance storage, such that both systems have become very similar in terms of use cases that they can cover. From an operational point of view it makes a lot of sense to integrate the Leonhard Open cluster into Euler as this allows us to exploit synergies in the system management, software installation and support.

The Leonhard Open shares will be transferred to Euler. This migration happens in a transparent way and apart from the hostname (euler.ethz.ch instead of login.leonhard.ethz.ch) nothing will change for the users. Both software stacks feature the same toolchains (GCC 4.8.5, 6.3.0, 8.2.0 and Intel 18.0.1) and both clusters are running the same batch system. We already started to install packages on Euler that were before only available on Leonhard Open to make the migration of the workflows as seamless as possible. Still there will be some installations missing and we will be happy to assist you in porting your workflows from Leonhard Open to Euler.

Data transfer

We will take care of transferring the data that shareholders store in the work storage. You would need to transfer the data from your home and personal scratch directory.

Schedule

We already migrated a few shareholders to Euler and plan to migrate all remaining shareholder on the 14 September 2021. This requires a one day downtime on Leonhard Open to do a final synchronization of the data from the work storage shares. On the 15 September after noon your data from Leonhard Open should be available on Euler.

FAQ

What happens to my Leonhard Open share?

Nothing. You will get exactly the same resources that you had on Leonhard Open on Euler for the remaining part of the 4 years duration that a share is valid. Since the Euler cluster is much larger than the Leonhard Open cluster, there will be a larger elasticity allowing for higher peak usage.

Do I need to transfer my data to Euler?

You will need to transfer data from your home directory and your personal scratch directory. We will take care of transferring the data from your work storage shares.

Is all the software that was available on Leonhard Open also available on Euler?

To prepare the integration of Leonhard Open into Euler we already started some months ago to identify differences in the software stacks of Leonhard Open and Euler and to install missing software packages on Euler that were only available on Leonhard Open. There will still be some packages missing that we can install again on request.