Setting up a software stack for research group

Introduction

Research groups that own a storage share in any of the HPC cluster of ETH have the possibility to setup their own software stack, which can then be shared within the group. They can also setup their own modules to manage such a software stack.

Example

We recommend that you create the subdirectories apps and modules in your storage share. In the apps directories, you would then manage the software stack, whereas the modules subdirecotry would be used the manage your own modules.

Euler

On the Euler cluster, we recommend to use storage shares in /cluster/project as the NetApp file system can better handle small files than Lustre (/cluster/work).

[sfux@eu-login-11-ng ~]$ cd /cluster/project/__YOUR_SHARE__ 
[sfux@eu-login-11-ng __YOUR_SHARE__]$ mkdir apps modules
[sfux@eu-login-11-ng __YOUR_SHARE__]$ chown sfux:sfux-group /cluster/project/__YOUR_SHARE__/apps /cluster/project/__YOUR_SHARE__/modules

Here you would need to replace __YOUR_SHARE__ by the name of your project share and sfux and sfux-group with your account name and personal group.

Leonhard

In this example we use a home directory as top-level directory for the software stack

[sfux@lo-login-01 ~]$ cd $HOME 
[sfux@lo-login-01 ~]$ mkdir apps modules
[sfux@lo-login-01 ~]$ chown sfux:sfux-group $HOME/apps $HOME/modules

Here you would need to replace sfux and sfux-group with your account name and personal group (or any other IAM group that is suitable for this purpose).

Structure inside the apps directory

We recommend that you use the following naming scheme to manage the applications in your software stack:

apps/application1/version1/architecture
apps/application1/version2/architecture

If you plan to install your own version of Python, such a structure could look like:

apps/python/2.7.9/x86_64
apps/python/2.7.11/x86_64

This is also the naming scheme that we use in /cluster/apps