Difference between revisions of "Modules and applications"

From ScientificComputing
Jump to: navigation, search
 
(45 intermediate revisions by 2 users not shown)
Line 1: Line 1:
 
__NOTOC__
 
__NOTOC__
 +
<table style="width: 100%;">
 +
<tr valign=top>
 +
<td style="width: 30%; text-align:left">
 +
< [[Storage and data transfer]]
 +
</td>
 +
<td style="width: 35%; text-align:center">
 +
[[Main_Page | Home]]
 +
</td>
 +
<td style="width: 35%; text-align:right">
 +
[[Job management with LSF]] >
 +
</td>
 +
</tr>
 +
</table>
 +
 +
 
<table style="width: 100%">
 
<table style="width: 100%">
 
<tr valign=center>
 
<tr valign=center>
 
<td style="width: 40%; background: white; ">
 
<td style="width: 40%; background: white; ">
[[Image:software_stack.png|center|300px]]
+
[[Image:environment.png|500px]]
  
 
</td>
 
</td>
<td style="width: 2%; background: white;">
+
<td style="width: 5%; background: white;">
 
</td>
 
</td>
 
<td style="width: 55%; background: white; text-align=left;">
 
<td style="width: 55%; background: white; text-align=left;">
Several software tools, each with several versions, are offered on the cluster. How to find applications you need and set up the computing environment specific for you?
+
On the cluster, we provide many centrally installed software and for some software even multiple versions. To configure the environment for a particular software version, we use modules.
 +
Modules configure your current computing environment (PATH, LD_LIBRARY_PATH, MANPATH, etc.) to make sure all required binaries and libraries are found.
  
'''Use Modules'''
+
We employ two types of Modules packages on the cluster:
 
+
* '''LMOD Modules'''  
A Modules package is a tool to let you to easily configure your computing environment.
+
* '''Environment Modules'''  
 
 
We employ two types of Modules packages on the clusters:
 
* '''LMOD Modules''' implemented on Euler and Leonhard Open
 
* '''Environment Modules''' implemented on Euler
 
  
 
</td>
 
</td>
Line 23: Line 35:
 
</table>
 
</table>
  
== Modules commands ==
+
== Two software stacks on Euler ==
Here are the most frequently used module commands.
 
  
 
<table style="width: 100%;">
 
<table style="width: 100%;">
 
<tr valign=top>
 
<tr valign=top>
<td style="width: 40%; ">
+
<td style="width: 45%;">
List all modules that match ''name''
+
=== Old software stack with Environment Modules ===
module avail ''name''
+
 
 +
Upon your login on Euler, the old software stack is set by default. <br />
 +
Switch from the old to the new software stack with the command:
 
</td>
 
</td>
 
<td style="width: 5%;">
 
<td style="width: 5%;">
 
</td>
 
</td>
<td style="width: 55%;">
+
<td style="width: 50%;">
Example: list all available Python
+
 
$ module avail python
+
=== New software stack with LMOD Modules ===
 +
All new software is installed exclusively in the new software stack, mostly done with [[New_SPACK_software_stack_on_Euler|SPACK package manager]].
 +
Switch from the new to old software stack:
 
</td>
 
</td>
 
</tr>
 
</tr>
  
 
<tr valign=top>
 
<tr valign=top>
<td style="width: 40%; ">
+
<td style="width: 45%; background: white;">
Load modules ''name''
+
$ env2lmod
  module load ''name''
+
 
 +
In a job script in Bash, add this line before loading modules. See an [[Job_submission#Job_script_and_.23BSUB_pragmas|example here]].
 +
 
 +
  source /cluster/apps/local/env2lmod.sh
 
</td>
 
</td>
<td style="width: 5%;">
+
<td style="width: 5%; background: white;">
 
</td>
 
</td>
<td style="width: 55%;">
+
<td style="width: 50%; background: white;">
Example: load GCC 6.3.0 and Python 3.8.5
+
  $ lmod2env
  $ module load gcc/6.3.0 python/3.8.5
 
 
</td>
 
</td>
 
</tr>
 
</tr>
 +
</table>
 +
 +
You can [[Setting_permanent_default_for_software_stack_upon_login|set the default software stack]] to the new one with the command:
 +
$ set_software_stack.sh new
 +
 +
Please note that after setting the default software stack, you need to logout and login again to make the change becoming active.
 +
 +
== Modules commands ==
 +
Here are the most frequently used module commands.
 +
 +
List all modules that match the given module name, e.g., list all available Python
 +
$ module avail python
 +
 +
Load modules, e.g., load GCC 6.3.0 and Python 3.8.5 in the new software stack. Please see [[Modules and applications#Two software stacks on Euler|Two software stacks on Euler]] for more details.
 +
$ env2lmod
 +
$ module load gcc/6.3.0 python/3.8.5
  
<tr valign=top>
 
<td style="width: 40%; ">
 
 
List all currently loaded modules
 
List all currently loaded modules
module list
 
</td>
 
<td style="width: 5%;">
 
</td>
 
<td style="width: 55%;">
 
Example:
 
 
  $ module list
 
  $ module list
 
   
 
   
 
  Currently Loaded Modules:
 
  Currently Loaded Modules:
 
   1) StdEnv  2) gcc/6.3.0  3) openblas/0.2.20  4) python/3.8.5
 
   1) StdEnv  2) gcc/6.3.0  3) openblas/0.2.20  4) python/3.8.5
</td>
+
 
</tr>
 
</table>
 
  
 
Other commonly used module commands
 
Other commonly used module commands
Line 78: Line 101:
 
  module unload ''name''        # unload module ''name''
 
  module unload ''name''        # unload module ''name''
 
  module purge              # unload all modules at once
 
  module purge              # unload all modules at once
 +
 +
For the new software stack only
 +
module spider            # report all modules
 +
module spider ''name''        # report all versions of the module ''name''
  
 
== Where are applications installed? ==  
 
== Where are applications installed? ==  
Line 97: Line 124:
 
=== In $HOME ===
 
=== In $HOME ===
 
Users can install additional applications in their home directory, but only if the quotas (space: 16GB, files/directories: 100'000) are not exceeded
 
Users can install additional applications in their home directory, but only if the quotas (space: 16GB, files/directories: 100'000) are not exceeded
* Avoid anaconda installations as they often conflict with the files/directories quota
+
* Avoid anaconda installations as they often conflict with the files/directories quota. Alternatively, you can create a [[Python virtual environment]].
* For Python and R, packages can easily be installed locally
+
* For Python and R, packages can easily be installed locally, e.g., for Python
 +
$ pip install --user packagename
 
</td>
 
</td>
 
</tr>
 
</tr>
 
</table>
 
</table>
  
== Two software stacks on Euler ==
+
== The structure of LMOD Modules ==  
Use the new software stack whenever possible
+
<table style="width: 100%; >
 
 
<table style="width: 100%;">
 
 
<tr valign=top>
 
<tr valign=top>
 
<td style="width: 45%;">
 
<td style="width: 45%;">
=== Old software stack with Env Modules ===
+
LMOD Modules use a hierarchy of modules with three layers to avoid conflicts when multiple modules are loaded at the same time.
'''Only on Euler'''
 
 
 
Upon your login on Euler, the old software stack is set by default. <br />
 
Switch from the old to the new software stack with the command:
 
</td>
 
<td style="width: 5%;">
 
</td>
 
<td style="width: 50%;">
 
 
 
=== New software stack with LMOD Modules ===
 
'''Available on Euler and Leonhard Open'''
 
 
 
 
 
Switch from the new to old software stack:
 
</td>
 
</tr>
 
 
 
<tr valign=top>
 
<td style="width: 45%; background: white;">
 
$ env2lmod
 
</td>
 
<td style="width: 5%; background: white;">
 
</td>
 
<td style="width: 50%; background: white;">
 
$ lmod2env
 
</td>
 
</tr>
 
</table>
 
 
 
All new software is installed exclusively in the new software stack, mostly done with [[New_SPACK_software_stack_on_Euler|SPACK package manager]].
 
 
 
You can find instructions how to load modules and examples on application pages, for instance, [[R]], [[Python]], [[MATLAB|Matlab]], etc.
 
 
 
== The structure of LMOD Modules ==
 
<table>
 
<tr valign=top>
 
<td style="width: 40%; background: white;">
 
LMOD Modules comprise of three layers
 
 
* The core layer contains software which are independent of compilers and MPI libraries, e.g., commercials software which come with their own runtime libraries
 
* The core layer contains software which are independent of compilers and MPI libraries, e.g., commercials software which come with their own runtime libraries
  module load comsol/5.6
+
  $ module load comsol/5.6
  
 
* The compiler layer contains software which are dependent of compilers.
 
* The compiler layer contains software which are dependent of compilers.
  module load gcc/6.3.0 hdf5/1.10.1
+
  $ module load gcc/6.3.0 hdf5/1.10.1
 
* The MPI layer contains software which are dependent of compilers and MPI libraries
 
* The MPI layer contains software which are dependent of compilers and MPI libraries
  module load gcc/6.3.0 openmpi/3.0.1 openblas
+
  $ module load gcc/6.3.0 openmpi/4.0.2 openblas
  
 
</td>
 
</td>
 
<td style="width: 5%; background: white;">
 
<td style="width: 5%; background: white;">
 
</td>
 
</td>
<td style="width: 55%; background: white;">
+
<td style="width: 50%; background: white;">
 
[[File:lmod_toolchains.png|400px]]
 
[[File:lmod_toolchains.png|400px]]
  
 
There are four main toolchains
 
There are four main toolchains
* GCC 4.8.5 (supports C++11 standard)
+
* [[Euler_applications_and_libraries#GCC_4.8.5|GCC 4.8.5]] (supports C++11 standard)
* GCC 6.3.0 (supports C++14 standard)
+
* [[Euler_applications_and_libraries#GCC_6.3.0|GCC 6.3.0]] (supports C++14 standard)
* GCC 8.2.0 (supports C++17 standard)
+
* [[Euler_applications_and_libraries#GCC_8.2.0|GCC 8.2.0]] (supports C++17 standard)
* Intel 18.0.1
+
* [[Euler_applications_and_libraries#Intel_19.1.0|Intel 19.1.0]]
 
Those compilers can be combined with OpenMPI 3.0.1 or 4.0.2 and OpenBLAS
 
Those compilers can be combined with OpenMPI 3.0.1 or 4.0.2 and OpenBLAS
  
Line 172: Line 160:
 
</tr>
 
</tr>
 
</table>
 
</table>
 +
 +
== Modules for GPUs ==
 +
In the new software stack, we have installed a Python module which is linked with CUDA, cuDNN and NCCL libraries and contains machine learning / deep learning packages such as scikit-Learn, TensorFlow and Pytorch. This module can be loaded with the following commands:
 +
$ env2lmod
 +
$ module load gcc/6.3.0 python_gpu/3.8.5
 +
 +
The following have been reloaded with a version change:
 +
  1) gcc/4.8.5 => gcc/6.3.0
 +
 +
$ module list
 +
 +
Currently Loaded Modules:
 +
  1) StdEnv            4) cuda/11.0.3    7) python_gpu/3.8.5
 +
  2) gcc/6.3.0        5) cudnn/8.0.5
 +
  3) openblas/0.2.20  6) nccl/2.7.8-1
 +
 +
You can also find and load CUDA, cuDNN and NCCL libraries available on the cluster matching your needs.
 +
  $ env2lmod
 +
  $ module load gcc/6.3.0
 +
  $ module avail cuda
 +
 +
------------- /cluster/apps/lmodules/Compiler/gcc/6.3.0 -------------
 +
  cuda/8.0.61    cuda/9.2.88      cuda/11.0.3 (L)
 +
  cuda/9.0.176    cuda/10.0.130    cuda/11.1.1
 +
  cuda/9.1.85    cuda/10.1.243    cuda/11.2.2 (D)
  
 
== Application lists ==
 
== Application lists ==
 
* [[Euler applications and libraries|Euler - New software stack]]
 
* [[Euler applications and libraries|Euler - New software stack]]
 
* [[Application life-cycle|Euler - Old software stack]]
 
* [[Application life-cycle|Euler - Old software stack]]
* [[Leonhard applications and libraries|Leonhard Open]]
 
 
* [[Main Page|Most recently installed software]]
 
* [[Main Page|Most recently installed software]]
  
Line 209: Line 221:
 
|-  
 
|-  
 
| GCC<br />Intel
 
| GCC<br />Intel
|| C, C++<br />Fortran, Go<br />Java, Julia<br />Perl, Python<br />Ruby, Scal
+
|| C, C++<br />Fortran, Go<br />Java, Julia<br />Perl, Python<br />Ruby, Scala
 
|| Boost, Eigen<br />FFTW, GMP<br />GSL, HDF5<br />MKL, NetCDF<br />NumPy, OpenBLAS<br />SciPy
 
|| Boost, Eigen<br />FFTW, GMP<br />GSL, HDF5<br />MKL, NetCDF<br />NumPy, OpenBLAS<br />SciPy
|| PETSc<br />Gurobi<br />Hypre<br />Trilino
+
|| PETSc<br />Gurobi<br />Hypre<br />Trilinos
 
|| Open MPI<br />Intel MPI<br />MPICH
 
|| Open MPI<br />Intel MPI<br />MPICH
 
|| GNU Autotools<br />Cmake<br />qmake<br />make
 
|| GNU Autotools<br />Cmake<br />qmake<br />make
Line 218: Line 230:
  
 
== Example ==  
 
== Example ==  
In this example we load GCC compiler version 4.8.2 and the most recent Python version in the old software stack, switch to the new software stack where we load GCC 6.3.0 and the most recent Python version, then switch back to the old software stack to load Python again
+
* [[Python virtual environment]]
 
 
Load GCC and new module in the old software stack:
 
$ module load new gcc/4.8.2
 
$ module avail python
 
 
------------------------------ /cluster/apps/modules/modulefiles ------------------------------
 
python/2.7.12  python/2.7.13  python/2.7.6(2.7)  python/2.7.9  python/3.3.3(3:3.3)  python/3.4.3      python/3.6.0
 
 
------------------------------ /cluster/apps/modules/new --------------------------------------
 
python/2.7.14    python/2.7.6_UCS4 python/3.6.1      python/3.7.1
 
 
 
Load the newest Python
 
$ module load python/3.7.1
 
Autoloading openblas/0.2.13_seq
 
 
 
$ module list
 
Currently Loaded Modulefiles:
 
  1) modules                            3) gcc/4.8.2(default:4.8)            5) python/3.7.1
 
  2) new                                4) openblas/0.2.13_seq(default:seq)
 
 
 
Switching to the new software stack:
 
$ env2lmod
 
$ module list 
 
 
Currently Loaded Modules:
 
  1) StdEnv  2) gcc/4.8.5
 
 
 
Load GCC 6.3.0 and check Python version
 
$ module load gcc/6.3.0
 
 
The following have been reloaded with a version change:
 
  1) gcc/4.8.5 => gcc/6.3.0
 
 
 
$ module avail python
 
 
--------------------------- /cluster/apps/lmodules/Compiler/gcc/6.3.0 -------------------------------------
 
  python/2.7.14    python/3.6.4    python/3.7.4    python/3.8.5 (D)    python_gpu/3.8.5
 
 
  Where:
 
  D:  Default Module
 
 
Use "module spider" to find all possible modules.
 
Use "module keyword key1 key2 ..." to search for all possible modules matching any of the "keys".
 
 
 
Load new Python
 
$ module load python/3.8.5
 
$ module list
 
 
Currently Loaded Modules:
 
  1) StdEnv  2) gcc/6.3.0  3) openblas/0.2.20  4) python/3.8.5
 
 
Switching back to the old software stack:
 
$ lmod2env
 
$ module list
 
Currently Loaded Modulefiles:
 
  1) modules
 
 
 
Load Python 3.7.1 again
 
$ module load new gcc/4.8.2 python/3.7.1
 
Autoloading openblas/0.2.13_seq
 
$ module list
 
Currently Loaded Modulefiles:
 
  1) modules                            3) gcc/4.8.2(default:4.8)            5) python/3.7.1
 
  2) new                                4) openblas/0.2.13_seq(default:seq)
 
  
 
== Further reading ==  
 
== Further reading ==  
Line 289: Line 237:
 
* [[Creating_local_module_directory|Creating a local module directory]]
 
* [[Creating_local_module_directory|Creating a local module directory]]
 
* [[Unpacking_RPM_packages|Unpacking RPM packages in users space]]
 
* [[Unpacking_RPM_packages|Unpacking RPM packages in users space]]
 +
 +
 +
<table style="width: 100%;">
 +
<tr valign=top>
 +
<td style="width: 30%; text-align:left">
 +
< [[Storage and data transfer]]
 +
</td>
 +
<td style="width: 35%; text-align:center">
 +
[[Main_Page | Home]]
 +
</td>
 +
<td style="width: 35%; text-align:right">
 +
[[Job management with LSF]] >
 +
</td>
 +
</tr>
 +
</table>

Latest revision as of 14:53, 21 December 2021

< Storage and data transfer

Home

Job management with LSF >


Environment.png

On the cluster, we provide many centrally installed software and for some software even multiple versions. To configure the environment for a particular software version, we use modules. Modules configure your current computing environment (PATH, LD_LIBRARY_PATH, MANPATH, etc.) to make sure all required binaries and libraries are found.

We employ two types of Modules packages on the cluster:

  • LMOD Modules
  • Environment Modules

Two software stacks on Euler

Old software stack with Environment Modules

Upon your login on Euler, the old software stack is set by default.
Switch from the old to the new software stack with the command:

New software stack with LMOD Modules

All new software is installed exclusively in the new software stack, mostly done with SPACK package manager. Switch from the new to old software stack:

$ env2lmod

In a job script in Bash, add this line before loading modules. See an example here.

source /cluster/apps/local/env2lmod.sh
$ lmod2env

You can set the default software stack to the new one with the command:

$ set_software_stack.sh new

Please note that after setting the default software stack, you need to logout and login again to make the change becoming active.

Modules commands

Here are the most frequently used module commands.

List all modules that match the given module name, e.g., list all available Python

$ module avail python

Load modules, e.g., load GCC 6.3.0 and Python 3.8.5 in the new software stack. Please see Two software stacks on Euler for more details.

$ env2lmod
$ module load gcc/6.3.0 python/3.8.5

List all currently loaded modules

$ module list

Currently Loaded Modules:
 1) StdEnv   2) gcc/6.3.0   3) openblas/0.2.20   4) python/3.8.5


Other commonly used module commands

module                    # get info about module sub-commands
module avail              # List all modules available on the cluster
module key keyword        # list all modules whose description contains keyword
module help name          # get information about module name
module show name          # show what module name does (without loading it)
module unload name        # unload module name
module purge              # unload all modules at once

For the new software stack only

module spider             # report all modules
module spider name        # report all versions of the module name

Where are applications installed?

Centrally installed

Central applications are installed in /cluster/apps

Applications that are needed by many users should be installed centrally, like compilers and libraries

  • Visible and accessible to all users via modules
  • Installed and maintained by cluster support
  • Commercial licenses provided by the IT shop of ETH or by research groups

In $HOME

Users can install additional applications in their home directory, but only if the quotas (space: 16GB, files/directories: 100'000) are not exceeded

  • Avoid anaconda installations as they often conflict with the files/directories quota. Alternatively, you can create a Python virtual environment.
  • For Python and R, packages can easily be installed locally, e.g., for Python
$ pip install --user packagename

The structure of LMOD Modules

LMOD Modules use a hierarchy of modules with three layers to avoid conflicts when multiple modules are loaded at the same time.

  • The core layer contains software which are independent of compilers and MPI libraries, e.g., commercials software which come with their own runtime libraries
$ module load comsol/5.6
  • The compiler layer contains software which are dependent of compilers.
$ module load gcc/6.3.0 hdf5/1.10.1
  • The MPI layer contains software which are dependent of compilers and MPI libraries
$ module load gcc/6.3.0 openmpi/4.0.2 openblas

Lmod toolchains.png

There are four main toolchains

Those compilers can be combined with OpenMPI 3.0.1 or 4.0.2 and OpenBLAS

Modules for GPUs

In the new software stack, we have installed a Python module which is linked with CUDA, cuDNN and NCCL libraries and contains machine learning / deep learning packages such as scikit-Learn, TensorFlow and Pytorch. This module can be loaded with the following commands:

$ env2lmod
$ module load gcc/6.3.0 python_gpu/3.8.5

The following have been reloaded with a version change:
  1) gcc/4.8.5 => gcc/6.3.0

$ module list

Currently Loaded Modules:
  1) StdEnv            4) cuda/11.0.3    7) python_gpu/3.8.5
  2) gcc/6.3.0         5) cudnn/8.0.5
  3) openblas/0.2.20   6) nccl/2.7.8-1

You can also find and load CUDA, cuDNN and NCCL libraries available on the cluster matching your needs.

 $ env2lmod
 $ module load gcc/6.3.0
 $ module avail cuda

------------- /cluster/apps/lmodules/Compiler/gcc/6.3.0 -------------
  cuda/8.0.61     cuda/9.2.88      cuda/11.0.3 (L)
  cuda/9.0.176    cuda/10.0.130    cuda/11.1.1
  cuda/9.1.85     cuda/10.1.243    cuda/11.2.2 (D)

Application lists

Table: Examples of centrally-installed commercial and open source applications
Bioinformatics & life science Finite element methods Machine learning Multi-physics phenomena Quantum chemistry & molecular dynamics Symbolic, numerical and statistical mathematics Visualization
Bamtools
BLAST
Bowtie
CLC Genomics Server
RAxML
Relion
TopHat
Ansys
Abaqus
FEniCS
PyTorch
Scikit-Learn
TensorFlow
Theano
AnsysEDT
COMSOL Multiphysics
STAR-CCM+
ADF
Ambertools
Gaussian
Molcas
Octopus
Orca
Qbox
Turbomole
Gurobi
Mathematica
MATLAB
R
Stata
Ffmpeg
ParaView
VisIT
VTK
Table: Examples of centrally-installed software development tools
Compiler Programming languages Scientific libraries Solvers MPI libraries Build systems Version control
GCC
Intel
C, C++
Fortran, Go
Java, Julia
Perl, Python
Ruby, Scala
Boost, Eigen
FFTW, GMP
GSL, HDF5
MKL, NetCDF
NumPy, OpenBLAS
SciPy
PETSc
Gurobi
Hypre
Trilinos
Open MPI
Intel MPI
MPICH
GNU Autotools
Cmake
qmake
make
CVS
Git
Mercurial
SVN

Example

Further reading


< Storage and data transfer

Home

Job management with LSF >