Modules and applications

From ScientificComputing
Revision as of 13:23, 10 May 2021 by Jarunanp (talk | contribs)

Jump to: navigation, search

Environment.png

Several software tools, each with several versions, are offered on the cluster. How to find applications you need and set up the computing environment specific for you?

Use Modules

A Modules package is a tool to let you to easily configure your computing environment.

We employ two types of Modules packages on the clusters:

  • LMOD Modules implemented on Euler and Leonhard Open
  • Environment Modules implemented on Euler

Modules commands

Here are the most frequently used module commands.

List all modules that match the given module name, e.g., list all available Python

module avail python

Load modules, e.g., load GCC 6.3.0 and Python 3.8.5

module load gcc/6.3.0 python/3.8.5

List all currently loaded modules

$ module list

Currently Loaded Modules:
 1) StdEnv   2) gcc/6.3.0   3) openblas/0.2.20   4) python/3.8.5


Other commonly used module commands

module                    # get info about module sub-commands
module avail              # List all modules available on the cluster
module key keyword        # list all modules whose description contains keyword
module help name          # get information about module name
module show name          # show what module name does (without loading it)
module unload name        # unload module name
module purge              # unload all modules at once

Where are applications installed?

Centrally installed

Central applications are installed in /cluster/apps

Applications that are needed by many users should be installed centrally, like compilers and libraries

  • Visible and accessible to all users via modules
  • Installed and maintained by cluster support
  • Commercial licenses provided by the IT shop of ETH or by research groups

In $HOME

Users can install additional applications in their home directory, but only if the quotas (space: 16GB, files/directories: 100'000) are not exceeded

  • Avoid anaconda installations as they often conflict with the files/directories quota
  • For Python and R, packages can easily be installed locally

Two software stacks on Euler

Use the new software stack whenever possible

Old software stack with Env Modules

Only on Euler

Upon your login on Euler, the old software stack is set by default.
Switch from the old to the new software stack with the command:

New software stack with LMOD Modules

Available on Euler and Leonhard Open


Switch from the new to old software stack:

$ env2lmod
$ lmod2env

All new software is installed exclusively in the new software stack, mostly done with SPACK package manager.

You can find instructions how to load modules and examples on application pages, for instance, R, Python, Matlab, etc.

The structure of LMOD Modules

LMOD Modules comprise of three layers

  • The core layer contains software which are independent of compilers and MPI libraries, e.g., commercials software which come with their own runtime libraries
module load comsol/5.6
  • The compiler layer contains software which are dependent of compilers.
module load gcc/6.3.0 hdf5/1.10.1
  • The MPI layer contains software which are dependent of compilers and MPI libraries
module load gcc/6.3.0 openmpi/3.0.1 openblas

Lmod toolchains.png

There are four main toolchains

  • GCC 4.8.5 (supports C++11 standard)
  • GCC 6.3.0 (supports C++14 standard)
  • GCC 8.2.0 (supports C++17 standard)
  • Intel 18.0.1

Those compilers can be combined with OpenMPI 3.0.1 or 4.0.2 and OpenBLAS

Application lists

Table: Examples of centrally-installed commercial and open source applications
Bioinformatics & life science Finite element methods Machine learning Multi-physics phenomena Quantum chemistry & molecular dynamics Symbolic, numerical and statistical mathematics Visualization
Bamtools
BLAST
Bowtie
CLC Genomics Server
RAxML
Relion
TopHat
Ansys
Abaqus
FEniCS
PyTorch
Scikit-Learn
TensorFlow
Theano
AnsysEDT
COMSOL Multiphysics
STAR-CCM+
ADF
Ambertools
Gaussian
Molcas
Octopus
Orca
Qbox
Turbomole
Gurobi
Mathematica
MATLAB
R
Stata
Ffmpeg
ParaView
VisIT
VTK
Table: Examples of centrally-installed software development tools
Compiler Programming languages Scientific libraries Solvers MPI libraries Build systems Version control
GCC
Intel
C, C++
Fortran, Go
Java, Julia
Perl, Python
Ruby, Scal
Boost, Eigen
FFTW, GMP
GSL, HDF5
MKL, NetCDF
NumPy, OpenBLAS
SciPy
PETSc
Gurobi
Hypre
Trilino
Open MPI
Intel MPI
MPICH
GNU Autotools
Cmake
qmake
make
CVS
Git
Mercurial
SVN

Example: Python virtual environment

This example shows how to create a Python virtual environment. First, switch to the new software stack

$ env2lmod

or, set your default software stack to the new software stack

$ set_software_stack.sh new

Load a Python module

$ module load gcc/6.3.0 python/3.8.5

Create a Python module

$ python -m venv –-system-site-packages myenv

Activate the virtual environment

$ source myenv/bin/activate

Check the python packages included in this environment

(myenv)$ pip list

You can install new packages inside this environment

(myenv)$ pip install --upgrade pip

Deactivate the Python virtual environment

(myenv)$ deactivate

Further reading


< Storage and data transfer

Job management with LSF >