Difference between revisions of "Modules and applications"

From ScientificComputing
Jump to: navigation, search
Line 132: Line 132:
 
</table>
 
</table>
  
−
All new software is installed exclusively in the new software stack. Many of the application pages (for instance [[R]], [[Python]], [[MATLAB|Matlab]], etc.) contain important information that is also valid for the installations in the new software stack.
+
All new software is installed exclusively in the new software stack, mostly done with [[New_SPACK_software_stack_on_Euler|SPACK package manage]]. Many of the application pages (for instance [[R]], [[Python]], [[MATLAB|Matlab]], etc.) contain important information that is also valid for the installations in the new software stack.
  
 
== The structure of LMOD Modules ==  
 
== The structure of LMOD Modules ==  

Revision as of 13:07, 28 January 2021

300px

What is a Modules package?

A Modules package is a tool to let users to easily configure their computing environment which includes

  • Software development tools
  • Scientific libraries
  • Communication libraries (MPI)
  • Third-party applications

We employ two types of Modules packages:

  • LMOD Modules implemented on Euler and Leonhard Open
  • Environment Modules implemented on Euler

Modules commands

Here are the most frequently used module commands.

List all modules that match name 

module avail name

Example: list all available Python 

$ module avail python

Load modules name 

module load name

Example: load GCC 6.3.0 and Python 3.8.5 

$ module load gcc/6.3.0 python/3.8.5

List all currently loaded modules 

module list

Example: 

$ module list

Currently Loaded Modules:
 1) StdEnv   2) gcc/6.3.0   3) openblas/0.2.20   4) python/3.8.5

Other commonly used module commands 

module                    # get info about module sub-commands
module avail              # List all modules available on the cluster
module key keyword        # list all modules whose description contains keyword
module help name          # get information about module name
module show name          # show what module name does (without loading it)
module unload name        # unload module name
module purge              # unload all modules at once

Where are applications installed?

Centrally installed

Central applications are installed in /cluster/apps

Applications that are needed by many users should be installed centrally, like compilers and libraries

  • Visible and accessible to all users via modules
  • Installed and maintained by cluster support
  • Commercial licenses provided by the IT shop of ETH or by research groups

In $HOME

Users can install additional applications in their home directory, but only if the quotas (space: 16GB, files/directories: 100'000) are not exceeded

  • Avoid anaconda installations as they often conflict with the files/directories quota
  • For Python and R, packages can easily be installed locall

Two software stacks on Euler

  • Use the new software stack whenever possible
  • Upon your login on Euler, the old software stack is set by default.

New software stack (LMOD Modules)

Available on Euler and Leonhard Open

Switch from the old to the new software stack with the command: 

$ env2lmod

Old software stack (Env Modules)

Only on Euler

and vice-versa 

$ lmod2env

All new software is installed exclusively in the new software stack, mostly done with SPACK package manage. Many of the application pages (for instance R, Python, Matlab, etc.) contain important information that is also valid for the installations in the new software stack.

The structure of LMOD Modules

To help you find and employ central applications more smoothly and efficiently, we explain here the structure and the toolchains of applications on the clusters.

Lmod toolchains.png

There are four main toolchains

  • GCC 4.8.5 (supports C++11 standard)
  • GCC 6.3.0 (supports C++14 standard)
  • GCC 8.2.0 (supports C++17 standard)
  • Intel 18.0.1

Those compilers can be combined with OpenMPI 3.0.1 or 4.0.2 and OpenBLAS

LMOD Modules comprise of three layers

  • The core layer contains software which are independent of compilers and MPI libraries, e.g., commercials software which come with their own runtime libraries
module load comsol/5.6
  • The compiler layer contains software which are dependent of compilers.
module load gcc/6.3.0 hdf5/1.10.1
  • The MPI layer contains software which are dependent of compilers and MPI libraries
module load gcc/6.3.0 openmpi/3.0.1 openblas

Application lists

Table: Examples of centrally-installed commercial and open source applications
Bioinformatics & life science Finite element methods Machine learning Multi-physics phenomena Quantum chemistry & molecular dynamics Symbolic, numerical and statistical mathematics Visualization
Bamtools
BLAST
Bowtie
CLC Genomics Server
RAxML
Relion
TopHat 		Ansys
Abaqus
FEniCS 		PyTorch
Scikit-Learn
TensorFlow
Theano 		AnsysEDT
COMSOL Multiphysics
STAR-CCM+ 		ADF
Ambertools
Gaussian
Molcas
Octopus
Orca
Qbox
Turbomole 		Gurobi
Mathematica
MATLAB
R
Stata 		Ffmpeg
ParaView
VisIT
VTK
Table: Examples of centrally-installed software development tools
Compiler Programming languages Scientific libraries Solvers MPI libraries Build systems Version control
GCC
Intel 		C, C++
Fortran, Go
Java, Julia
Perl, Python
Ruby, Scal 		Boost, Eigen
FFTW, GMP
GSL, HDF5
MKL, NetCDF
NumPy, OpenBLAS
SciPy 		PETSc
Gurobi
Hypre
Trilino 		Open MPI
Intel MPI
MPICH 		GNU Autotools
Cmake
qmake
make 		CVS
Git
Mercurial
SVN

Example

In this example we load GCC compiler version 4.8.2 and the most recent Python version in the old software stack, switch to the new software stack where we load GCC 6.3.0 and the most recent Python version, then switch back to the old software stack to load Python again

Load GCC and new module in the old software stack: 

$ module load new gcc/4.8.2
$ module avail python

------------------------------ /cluster/apps/modules/modulefiles ------------------------------
python/2.7.12  python/2.7.13  python/2.7.6(2.7)   python/2.7.9  python/3.3.3(3:3.3)   python/3.4.3      python/3.6.0

------------------------------ /cluster/apps/modules/new --------------------------------------
python/2.7.14     python/2.7.6_UCS4 python/3.6.1      python/3.7.1

Load the newest Python 

$ module load python/3.7.1
Autoloading openblas/0.2.13_seq

$ module list
Currently Loaded Modulefiles:
  1) modules                            3) gcc/4.8.2(default:4.8)             5) python/3.7.1
  2) new                                4) openblas/0.2.13_seq(default:seq)

Switching to the new software stack: 

$ env2lmod
$ module list  

Currently Loaded Modules:
  1) StdEnv   2) gcc/4.8.5

Load GCC 6.3.0 and check Python version 

$ module load gcc/6.3.0

The following have been reloaded with a version change:
  1) gcc/4.8.5 => gcc/6.3.0

$ module avail python

--------------------------- /cluster/apps/lmodules/Compiler/gcc/6.3.0 -------------------------------------
  python/2.7.14    python/3.6.4    python/3.7.4    python/3.8.5 (D)    python_gpu/3.8.5

 Where:
  D:  Default Module

Use "module spider" to find all possible modules.
Use "module keyword key1 key2 ..." to search for all possible modules matching any of the "keys".

Load new Python 

$ module load python/3.8.5
$ module list

Currently Loaded Modules:
 1) StdEnv   2) gcc/6.3.0   3) openblas/0.2.20   4) python/3.8.5

Switching back to the old software stack: 

$ lmod2env
$ module list
Currently Loaded Modulefiles:
  1) modules

Load Python 3.7.1 again 

$ module load new gcc/4.8.2 python/3.7.1
Autoloading openblas/0.2.13_seq
$ module list
Currently Loaded Modulefiles:
  1) modules                            3) gcc/4.8.2(default:4.8)             5) python/3.7.1
  2) new                                4) openblas/0.2.13_seq(default:seq)

Further reading

Retrieved from "https://scicomp.ethz.ch/w/index.php?title=Modules_and_applications&oldid=5586"