New SPACK software stack on Euler

Introduction

On the Euler cluster, we are in the transition phase from the old software stack to the new software stack. Currently when users login, the old software stack is still set as the default. It will be kept on an as-is basis to allow reproducing older results. New software is only installed in the new software stack. It is set up with the package manager SPACK on Euler using the LMOD module system. The new software stack has 4 basic toolchains (GCC 4.8.5, GCC 6.3.0, GCC 8.2.0 and Intel 19.1.0). A toolchain is a combination of compiler, MPI and BLAS/LAPACK library. For each of the toolchains, several hundred packages are available.

Timeline

A few selected users (early beta testers) have been using the new software stack since end of August 2018. The open beta testing phase of the new software stack started on 10 December 2018. During 2019, a large part of the new software stack has been built (when counting all packages from all toolchains, currently 5413 packages are provided). A list of all software that is present in the old software stack but not yet in the new is provided here, including a separate list for the GDC software stack.

By default, new cluster users have the new software stack set as default. Users can switch between the old and the new software stack by using the commands env2lmod and lmod2env. New software installations will exclusively be done in the new software stack.

Available software

You can find an overview on all available packages in the new software stack on this wiki page.

Differences between the old and the new software stack

• The old software stack was mainly based on manual software installation, not using a package manager. Most packages were available for the standard compiler and only a limited subset of libraries was available for newer compilers as well. In the new software stack, we have put more emphasis on the concept of toolchains, providing most packages for all compilers.
• In the old software stacks, users often had to load modules for the dependencies of a main program. In the new software stack, it is sufficient to load the module of the main program and all dependencies will be loaded automatically (one exception is MPI, as MPI modules need to be loaded to access those branches of the toolchains).

Old software stack:
module load new gcc/4.8.2 open_mpi/1.6.5 boost/1.59.0_py2.7.9 mpfr/3.1.2_gmp6 qt/4.8.4 netcdf/4.3.2 eigen/3.2.1 swig/3.0.5 suitesparse/4.4.4 fenics/1.6.0 

New software stack:
module load gcc/6.3.0 openmpi/3.0.1 fenics/2019.1.0_slepc

• In the old software stack, it was possible to use packages compiled with a different compiler together, which often caused issues. In the new software stack, the module hierarchy ensures that only packages that were compiled with the same compiler version can be used together.

Hierarchical modules

LMOD allows to define a hierarchy of modules containing 3 layers (Core, Compiler, MPI). The core layer contains all module files which are not depending on any compiler/MPI. The compiler layer contains all modules which are depending on a particular compilers, but not on any MPI library. The MPI layer contains modules that are depending on a particular compiler/MPI combination.

When you login to the Leonhard cluster, the standard compiler gcc/4.8.5 is automatically loaded. Running the module avail command displays all modules that are available for gcc/4.8.5. If you would like to see the modules available for a different compiler, for instance gcc/6.3.0, then you would need to load the compiler module and run module avail again. For checking out the available modules for gcc/4.8.5 openmpi/3.0.1, you would load the corresponding compiler and MPI module and run again module avail.

As a consequence of the module hierarchy, you can never have two different versions of the same module loaded at the same time. This helps to avoid problems arising due to misconfiguration of the environment.

Switching between software stacks

It is possible to switch back and forth between the old and the new software stack, but they cannot be mixed. Please use the commands env2lmod and lmod2env for this. These are aliases that source a script that changes the environment of the current shell.

Switching from the old to the new software stack:

env2lmod

or

source /cluster/apps/local/env2lmod.sh

Switching from the new to the old software stack:

lmod2env

or

source /cluster/apps/local/lmod2env.sh

Changing the software stack will unload all previously loaded modules before the switch. When you login, then the old software stack is initialized by default. We now provide a script set_software_stack.sh that allows you to set a permanent default for the software stack (old/new) upon login.

Setting a permanent default for the software stack

You can set a permanent default for the software stack that should be initialized for your account upon login. You can choose between the old and the new software stacks. Please note that after setting a permanent default, you need to logout and login again to activate the change. You can change the default as often as you want.

Setting the new software stack as permanent default:

set_software_stack.sh new

Setting the old software stack as permanent default:

set_software_stack.sh old

Checking the current setting for the software stack that is set as default:

set_software_stack.sh -i

Example

Loading some modules in the old software stack:

[sfux@eu-login-16-ng ~]$ module load new gcc/4.8.2 open_mpi/1.6.5 openblas/0.2.8_seq
[sfux@eu-login-16-ng ~]$ module list
Currently Loaded Modulefiles:
  1) modules                       3) gcc/4.8.2(default:4.8)        5) openblas/0.2.8_seq(0.2.8)
  2) new                           4) open_mpi/1.6.5(1.6:default)

Switching to the new software stack and load some modules:

[sfux@eu-login-16-ng ~]$ env2lmod
[sfux@eu-login-16-ng ~]$ module list  

Currently Loaded Modules:
  1) StdEnv   2) gcc/4.8.5

  

[sfux@eu-login-16-ng ~]$ module load intel/18.0.1 openmpi/3.0.1 openblas/0.2.20

Lmod is automatically replacing "gcc/4.8.5" with "intel/18.0.1".

[sfux@eu-login-16-ng ~]$ 

Switching back to the old software stack and loading again some modules:

[sfux@eu-login-16-ng ~]$ lmod2env
[sfux@eu-login-16-ng ~]$ module list
Currently Loaded Modulefiles:
  1) modules
[sfux@eu-login-16-ng ~]$ module load new gcc/4.8.2 open_mpi/1.6.5 openblas/0.2.8_seq
[sfux@eu-login-16-ng ~]$ module list
Currently Loaded Modulefiles:
  1) modules                       3) gcc/4.8.2(default:4.8)        5) openblas/0.2.8_seq(0.2.8)
  2) new                           4) open_mpi/1.6.5(1.6:default)
[sfux@eu-login-16-ng ~]$