Difference between revisions of "Creating local module directory"

From ScientificComputing
Jump to: navigation, search
(Custom modules directory.)
 
(26 intermediate revisions by 3 users not shown)
Line 1: Line 1:
−
== Introduction ==
+
==Introduction==
 +
You can use your own custom modules to setup your environment to run programs that you have installed yourself. This tutorial will guide you through creating a custom module directory, adding versioned modules for programs, and automatically having these modules available to you upon login. The tutorial will assume you will be managing a set of modules for a whole group, though it is equally applicable to managing a personal set of modules.
  
−
You can use your own custom modules to setup your environment to run programs that you have installed yourself.
+
Please note that on Euler, we are using environment modules in the old software stack and LMOD modules in the new software stack. On Leonhard we are using LMOD modules. When extending the centrally provided modules, please make sure that you use the correct module type for your own modules.
  
−
This tutorial will guide you through creating a custom module directory, adding versioned modules for programs, and automatically having these modules available to you upon login. The tutorial will assume you will be managing a set of modules for a whole group, though it is equally applicable to managing a personal set of modules.
+
==Extending the old software stack on Euler (environment modules)==
 +
You need to decide on the top-level directory (we will use the placeholder __TLD__ for the examples) in which to store the module files. You need to create the directory and set the user and group ownership:
  
−
=== Creating a custom module directory ===
+
[sfux@eu-login-01-ng ~]$ '''cd __TLD__'''
 +
[sfux@eu-login-01-ng __TLD__]$ '''mkdir modules'''
 +
[sfux@eu-login-01-ng __TLD__]$ '''chown sfux:sfux-group modules'''
  
−
You need to decide on the top-level directory in which to store the module files. Let us assume it is <tt>/cluster/project/leonhard/modules</tt>, where <tt>hpc</tt> should be replaced with your group's storage location. This can also be an arbitrary directory to which you have access, e.g., <tt>$HOME/modules</tt>. First create the directory and set proper permissions:
+
Here you would need to replace <tt>sfux</tt> and <tt>sfux-group</tt> with your account name and personal group (or any other IAM group that is suitable for this purpose).
−
mkdir /cluster/project/leonhard/modules
 
−
chown leonhard:leonhard-group /cluster/project/leonhard/modules
 
  
−
==== Using the custom module directory ====
+
Now you need to let the <tt>module</tt> command know about your custom directory. Add the following line to your <tt>$HOME/.bash_profile</tt> file in case you are the only person using the additional modules. Or you can create a script that can be sourced in case multiple people use the additional modules.
  
−
Now you need to let the <tt>module</tt> command know about your custom directory. Add the following line to your <tt>$HOME/.bash_profile</tt> file:
+
  module use __TLD__/modules
−
  module use /cluster/project/leonhard/modules
 
−
The <tt>module</tt> command will now see your custom modules the next time you log in. You can also type this command into your current shell to have immediate access.
 
  
−
Every group members will need to do this setup step to see the modules.
+
==Extending the LMOD module hierarchy with local modules==
 +
Extending the LMOD module hierarchy is a bit more complex than extending a system of environment modules without hierarchy.
  
−
== Creating a versioned module file ==
+
===Euler===
 +
As described above, you would need to create a top level directory for your modules first (later we refer to this directory as __TLD__). Then download the [https://polybox.ethz.ch/index.php/s/PeAnZQg0WXE48xK local_modules.tar.gz] file and untar it in your top level directory. This will provide the skeleton of directories for the module hierarchy:
 +
 +
[sfux@eu-login-12 my_modules]$ tar -xzvf local_modules.tar.gz
 +
Compiler/
 +
Compiler/gcc/
 +
Compiler/gcc/8.2.0/
 +
Compiler/gcc/8.2.0/openmpi/
 +
Compiler/gcc/8.2.0/openmpi/3.0.1.lua
 +
Compiler/gcc/8.2.0/openmpi/4.0.2.lua
 +
Compiler/gcc/4.8.5/
 +
Compiler/gcc/4.8.5/openmpi/
 +
Compiler/gcc/4.8.5/openmpi/3.0.1.lua
 +
Compiler/gcc/4.8.5/openmpi/4.0.2.lua
 +
Compiler/gcc/6.3.0/
 +
Compiler/gcc/6.3.0/openmpi/
 +
Compiler/gcc/6.3.0/openmpi/3.0.1.lua
 +
Compiler/gcc/6.3.0/openmpi/4.0.2.lua
 +
Compiler/intel/
 +
Compiler/intel/18.0.1/
 +
Compiler/intel/18.0.1/openmpi/
 +
Compiler/intel/18.0.1/openmpi/3.0.1.lua
 +
Core/
 +
Core/gcc/
 +
Core/gcc/4.8.5.lua
 +
Core/gcc/8.2.0.lua
 +
Core/gcc/6.3.0.lua
 +
Core/intel/
 +
Core/intel/2018.1.lua
 +
Core/intel/18.0.1.lua
 +
MPI/
 +
MPI/openmpi/
 +
MPI/openmpi/3.0.1/
 +
MPI/openmpi/3.0.1/gcc/
 +
MPI/openmpi/3.0.1/gcc/8.2.0/
 +
MPI/openmpi/3.0.1/gcc/4.8.5/
 +
MPI/openmpi/3.0.1/gcc/6.3.0/
 +
MPI/openmpi/3.0.1/intel/
 +
MPI/openmpi/3.0.1/intel/18.0.1/
 +
MPI/openmpi/4.0.2/
 +
MPI/openmpi/4.0.2/gcc/
 +
MPI/openmpi/4.0.2/gcc/8.2.0/
 +
MPI/openmpi/4.0.2/gcc/4.8.5/
 +
MPI/openmpi/4.0.2/gcc/6.3.0/
 +
MPI/openmpi/4.0.2/intel/
 +
MPI/openmpi/4.0.2/intel/18.0.1/
 +
 
 +
Now you can start to add your self created modules into the hierarchy depending on what compiler/MPI library they depend on.
 +
 
 +
* Core layer: module files for software that does not depend on a centrally provided compiler or MPI library (self-contained binary release containing required runtime libraries, often commercial software like MATLAB, Ansys, Comsol, etc., or compilers themselves)
 +
* Compiler layer: modules for software that depends on a compiler but not on an MPI library
 +
* MPI layer: modules for software that depends on a particular compiler and MPI library
 +
 
 +
In order to use the local modules, you would need to use the following instructions:
 +
 
 +
* Login to Euler
 +
* Change to the new software stack, using the command env2lmod
 +
* export MY_MODULEPATH_ROOT=__TLD__
 +
* module use $MY_MODULEPATH_ROOT/Core
 +
 
 +
Please note that in the third step, __TLD__ needs to be replaced with the path to the top level directory of your local modules.
 +
 
 +
===Leonhard===
 +
This is work in progress and not yet documented.
 +
 
 +
===LMOD cache===
 +
LMOD uses a local cache to speed up '''module avail''' and '''module spider''' queries. Therefore if a new module file is created, you cannot immediately see it with '''module avail''' as it takes a while until the cache is refreshed, but you can already load it with '''module load''', because for loading a module the cache is not used to find the module.
 +
 
 +
If you would like to test a newly created module, then you can either remove the current cache files in
 +
 +
$HOME/.lmod.d/.cache
 +
 
 +
or you can use the option of the module command to bypass the cache:
 +
 +
module avail --ignore_cache
 +
 
 +
==Example module files==
 +
 
 +
===Environment modules===
 +
For each software that you would like to create a module for, create a directory in the <tt>modules</tt> directory. This directory then contains the file whose name is the version number. In this example, we use PROGRAM as placeholder and 1.0 as version. The module file would therefore be stored in <tt>__TLD__/modules/PROGRAM</tt> and have the name <tt>1.0</tt>
  
−
Next, you need to create a module file for a program. Assume you have version 1.0 of the <tt>basel</tt> program located in <tt>/cluster/project/leonhard/apps/basel_problem/1.0/bin</tt> and it depends on the <tt>gmp</tt> module. Create the program's module directory:
 
−
mkdir /cluster/project/leonhard/modules/basel
 
−
chown leonhard:leonhard-group /cluster/project/leonhard/modules/basel
 
−
In your favorite editor, edit the <tt>/cluster/project/leonhard/modules/basel/1.0</tt> file to contain the following contents:
 
 
  #%Module1.0
 
  #%Module1.0
−
  module-whatis "Solve Basel problem"
+
  module-whatis "This is my software PROGRAM version 1.0"
−
  set helpmsg "Solves the Basel problem."
+
  set helpmsg "PROGRAM is used used to achieve the following task ..."
−
module load gmp
+
  set topdir "__TLD__/apps/PROGRAM/1.0"
−
  set topdir "/cluster/project/leonhard/apps/basel_problem/1.0"
+
setenv IMPORTANT_VARIABLE "Some string"
 
  prepend-path PATH $topdir/bin
 
  prepend-path PATH $topdir/bin
 +
prepend-path LD_LIBRARY_PATH $topdir/lib64
 +
prepend-path C_INCLUDE_PATH $topdir/include
 +
 +
===LMOD modules===
 +
For each software that you would like to create a module for, create a directory in the <tt>modules</tt> directory. This directory then contains the file whose name is the version number followed by the ending <tt>.lua</tt>. In this example, we use PROGRAM as placeholder and 1.0 as version. The module file would therefore be stored in <tt>__TLD__/modules/PROGRAM</tt> and have the name <tt>1.0.lua</tt>
  
−
Test that it works:
+
<nowiki>
−
  module avail basel/1.0
+
-- -*- lua -*-
−
  module load basel/1.0
+
--
−
  module list
+
-- Any sort of comment can be put here
 +
--
 +
--
 +
whatis([[Name : This is my software PROGRAM version 1.0]])
 +
  help([[PROGRAM is used to achieve the following task ...]])
 +
setenv("IMPORTANT_VARIABLE", "Some string")
 +
prepend_path("PATH", "__TLD__/apps/PROGRAM/1.0/bin")
 +
  prepend_path("LD_LIBRARY_PATH", "__TLD__/apps/PROGRAM/1.0/lib64")
 +
  prepend_path("C_INCLUDE_PATH", "__TLD__/apps/PROGRAM/1.0/include")
 +
</nowiki>

Revision as of 07:17, 13 January 2022

Introduction

You can use your own custom modules to setup your environment to run programs that you have installed yourself. This tutorial will guide you through creating a custom module directory, adding versioned modules for programs, and automatically having these modules available to you upon login. The tutorial will assume you will be managing a set of modules for a whole group, though it is equally applicable to managing a personal set of modules.

Please note that on Euler, we are using environment modules in the old software stack and LMOD modules in the new software stack. On Leonhard we are using LMOD modules. When extending the centrally provided modules, please make sure that you use the correct module type for your own modules.

Extending the old software stack on Euler (environment modules)

You need to decide on the top-level directory (we will use the placeholder __TLD__ for the examples) in which to store the module files. You need to create the directory and set the user and group ownership: 

[sfux@eu-login-01-ng ~]$ cd __TLD__
[sfux@eu-login-01-ng __TLD__]$ mkdir modules
[sfux@eu-login-01-ng __TLD__]$ chown sfux:sfux-group modules

Here you would need to replace sfux and sfux-group with your account name and personal group (or any other IAM group that is suitable for this purpose).

Now you need to let the module command know about your custom directory. Add the following line to your $HOME/.bash_profile file in case you are the only person using the additional modules. Or you can create a script that can be sourced in case multiple people use the additional modules. 

module use __TLD__/modules

Extending the LMOD module hierarchy with local modules

Extending the LMOD module hierarchy is a bit more complex than extending a system of environment modules without hierarchy.

Euler

As described above, you would need to create a top level directory for your modules first (later we refer to this directory as __TLD__). Then download the local_modules.tar.gz file and untar it in your top level directory. This will provide the skeleton of directories for the module hierarchy: 

[sfux@eu-login-12 my_modules]$ tar -xzvf local_modules.tar.gz
Compiler/
Compiler/gcc/
Compiler/gcc/8.2.0/
Compiler/gcc/8.2.0/openmpi/
Compiler/gcc/8.2.0/openmpi/3.0.1.lua
Compiler/gcc/8.2.0/openmpi/4.0.2.lua
Compiler/gcc/4.8.5/
Compiler/gcc/4.8.5/openmpi/
Compiler/gcc/4.8.5/openmpi/3.0.1.lua
Compiler/gcc/4.8.5/openmpi/4.0.2.lua
Compiler/gcc/6.3.0/
Compiler/gcc/6.3.0/openmpi/
Compiler/gcc/6.3.0/openmpi/3.0.1.lua
Compiler/gcc/6.3.0/openmpi/4.0.2.lua
Compiler/intel/
Compiler/intel/18.0.1/
Compiler/intel/18.0.1/openmpi/
Compiler/intel/18.0.1/openmpi/3.0.1.lua
Core/
Core/gcc/
Core/gcc/4.8.5.lua
Core/gcc/8.2.0.lua
Core/gcc/6.3.0.lua
Core/intel/
Core/intel/2018.1.lua
Core/intel/18.0.1.lua
MPI/
MPI/openmpi/
MPI/openmpi/3.0.1/
MPI/openmpi/3.0.1/gcc/
MPI/openmpi/3.0.1/gcc/8.2.0/
MPI/openmpi/3.0.1/gcc/4.8.5/
MPI/openmpi/3.0.1/gcc/6.3.0/
MPI/openmpi/3.0.1/intel/
MPI/openmpi/3.0.1/intel/18.0.1/
MPI/openmpi/4.0.2/
MPI/openmpi/4.0.2/gcc/
MPI/openmpi/4.0.2/gcc/8.2.0/
MPI/openmpi/4.0.2/gcc/4.8.5/
MPI/openmpi/4.0.2/gcc/6.3.0/
MPI/openmpi/4.0.2/intel/
MPI/openmpi/4.0.2/intel/18.0.1/

Now you can start to add your self created modules into the hierarchy depending on what compiler/MPI library they depend on.

  • Core layer: module files for software that does not depend on a centrally provided compiler or MPI library (self-contained binary release containing required runtime libraries, often commercial software like MATLAB, Ansys, Comsol, etc., or compilers themselves)
  • Compiler layer: modules for software that depends on a compiler but not on an MPI library
  • MPI layer: modules for software that depends on a particular compiler and MPI library

In order to use the local modules, you would need to use the following instructions:

  • Login to Euler
  • Change to the new software stack, using the command env2lmod
  • export MY_MODULEPATH_ROOT=__TLD__
  • module use $MY_MODULEPATH_ROOT/Core

Please note that in the third step, __TLD__ needs to be replaced with the path to the top level directory of your local modules.

Leonhard

This is work in progress and not yet documented.

LMOD cache

LMOD uses a local cache to speed up module avail and module spider queries. Therefore if a new module file is created, you cannot immediately see it with module avail as it takes a while until the cache is refreshed, but you can already load it with module load, because for loading a module the cache is not used to find the module.

If you would like to test a newly created module, then you can either remove the current cache files in 

$HOME/.lmod.d/.cache

or you can use the option of the module command to bypass the cache: 

module avail --ignore_cache

Example module files

Environment modules

For each software that you would like to create a module for, create a directory in the modules directory. This directory then contains the file whose name is the version number. In this example, we use PROGRAM as placeholder and 1.0 as version. The module file would therefore be stored in __TLD__/modules/PROGRAM and have the name 1.0 

#%Module1.0
module-whatis "This is my software PROGRAM version 1.0"
set helpmsg "PROGRAM is used used to achieve the following task ..."
set topdir "__TLD__/apps/PROGRAM/1.0"
setenv IMPORTANT_VARIABLE "Some string"
prepend-path PATH $topdir/bin
prepend-path LD_LIBRARY_PATH $topdir/lib64
prepend-path C_INCLUDE_PATH $topdir/include

LMOD modules

For each software that you would like to create a module for, create a directory in the modules directory. This directory then contains the file whose name is the version number followed by the ending .lua. In this example, we use PROGRAM as placeholder and 1.0 as version. The module file would therefore be stored in __TLD__/modules/PROGRAM and have the name 1.0.lua 

 
 -- -*- lua -*-
 -- 
 -- Any sort of comment can be put here
 --
 --
 whatis([[Name : This is my software PROGRAM version 1.0]])
 help([[PROGRAM is used to achieve the following task ...]])
 setenv("IMPORTANT_VARIABLE", "Some string")
 prepend_path("PATH", "__TLD__/apps/PROGRAM/1.0/bin")
 prepend_path("LD_LIBRARY_PATH", "__TLD__/apps/PROGRAM/1.0/lib64")
 prepend_path("C_INCLUDE_PATH", "__TLD__/apps/PROGRAM/1.0/include")
Retrieved from "https://scicomp.ethz.ch/w/index.php?title=Creating_local_module_directory&oldid=6751"