Difference between revisions of "Leonhard beta testing"

From ScientificComputing
Jump to: navigation, search
(Storage)
 
(48 intermediate revisions by 3 users not shown)
Line 1: Line 1:
The [[Leonhard|Leonhard cluster]] is available for early-access beta testing.
+
{{LeoOpen_obsolete}}
  
Please read through the following to get started.
+
Please read through the following to get started on the [[Leonhard|Leonhard cluster]].
 +
 
 +
==Current status==
 +
The Leonhard Open cluster is still in the beta testing phase.
 +
 
 +
The '''lo-a2-*''', the '''lo-gtx-*''' and the '''lo-s4-*''' compute nodes are connected to the '''InfiniBand''' high-performance interconnect and can access the '''GPFS''' storage system (<tt>/cluster/scratch</tt>,<tt>/cluster/work</tt>) and the '''NetApp''' storage system (<tt>/cluster/project</tt>,<tt>/cluster/home</tt>).
  
 
==Accessing the cluster==
 
==Accessing the cluster==
Line 9: Line 14:
  
 
===SSH===
 
===SSH===
Users can access the Leonhard cluster via SSH.
+
You can find general information about how to access our clusters via SSH in the [[Accessing_the_clusters#SSH|Accessing the clusters]] tutorial.
 +
 
 +
The command to access the Leonhard cluster via SSH is:
  
 
  ssh ''username''@login.leonhard.ethz.ch
 
  ssh ''username''@login.leonhard.ethz.ch
  
where ''username'' corresponds to your NETHZ username.
+
where ''username'' corresponds to your ETH username.
  
==Using modules==
+
==Storage==
For the Leonhard cluster, we decided to switch from the [http://modules.sourceforge.net environment modules] that are used on the Euler cluster to [https://lmod.readthedocs.io/en/latest/index.html Lmod modules], which provide some nice features that are not available for environment modules. You should barely notice the transition from environment modules to Lmod modules as the commands are mostly the same:
+
You can find general information about the storage systems in the [[Storage_systems|Storage systems]] tutorial.
  
  [leonhard@lo-login-02 ~]$ module avail
+
Like on the Euler cluster, every user also has a home directory and a personal scratch directory:
 +
 
 +
/cluster/home/''username''
 +
/cluster/scratch/''username''
 +
 
 +
==Modules for preparing your environment==
 +
 
 +
===LMOD===
 +
For the Leonhard cluster, we decided to switch from the [http://modules.sourceforge.net environment modules] that are used on the Euler cluster to [https://lmod.readthedocs.io/en/latest/index.html Lmod modules], which provide some additional features. You should barely notice the transition from environment modules to Lmod modules as the commands are mostly the same. Therefore please refer to the [[Setting_up_your_environment|Setting up your environment]] tutorial for a general documentation about the module commands.
 +
 
 +
  [leonhard@lo-login-02 ~]$ '''module avail boost'''
 
   
 
   
  ------------------------------------------------- /cluster/spack/lmodules -------------------------------------------------
+
  ----------------------------------------- /cluster/apps/lmodules/Compiler/gcc/4.8.5 ------------------------------------------
     gcc/4.8.5/apr-util/1.5.4                        gcc/4.8.5/guile/2.0.11              gcc/4.8.5/lua/5.3.2
+
     boost/1.63.0
    gcc/4.8.5/apr/1.5.2                              gcc/4.8.5/help2man/1.47.4            gcc/4.8.5/lz4/1.7.5
+
    gcc/4.8.5/autoconf/2.69                          gcc/4.8.5/inputproto/2.3.2          gcc/4.8.5/lzma/4.32.7
+
Use "module spider" to find all possible modules.
    gcc/4.8.5/automake/1.15                          gcc/4.8.5/jpeg/9b                    gcc/4.8.5/lzo/2.09
+
Use "module keyword key1 key2 ..." to search for all possible modules matching any of the "keys".
    gcc/4.8.5/bash/4.4                              gcc/4.8.5/kbproto/1.0.7              gcc/4.8.5/m4/1.4.18
+
    gcc/4.8.5/bdw-gc/7.4.4                          gcc/4.8.5/lcms/2.8                  gcc/4.8.5/mawk/1.3.4
+
    gcc/4.8.5/binutils/2.28                          gcc/4.8.5/libarchive/3.2.1          gcc/4.8.5/mpfr/3.1.5
+
[leonhard@lo-login-02 ~]$ '''module load boost/1.63.0'''
    gcc/4.8.5/bison/3.0.4                            gcc/4.8.5/libatomic-ops/7.4.4        gcc/4.8.5/nasm/2.11.06
+
[leonhard@lo-login-02 ~]$ '''module list'''
    gcc/4.8.5/bzip2/1.0.6                            gcc/4.8.5/libctl/3.2.2              gcc/4.8.5/ncurses/6.0
+
    gcc/4.8.5/coreutils/8.26                        gcc/4.8.5/libdrm/2.4.70              gcc/4.8.5/nettle/3.2
+
Currently Loaded Modules:
    gcc/4.8.5/cscope/15.8b                          gcc/4.8.5/libdwarf/20160507          gcc/4.8.5/openblas/0.2.19
+
  1) gcc/4.8.5   2) StdEnv  3) boost/1.63.0
    gcc/4.8.5/cuda/8.0.61                            gcc/4.8.5/libelf/0.8.13              gcc/4.8.5/openssl/1.0.1e
+
    gcc/4.8.5/cudnn/6.0                              gcc/4.8.5/libffi/3.2.1              gcc/4.8.5/patch/2.7.5
+
[leonhard@lo-login-02 ~]$
    gcc/4.8.5/curl/7.53.1                            gcc/4.8.5/libfontenc/1.1.3          gcc/4.8.5/patchelf/0.9
 
    gcc/4.8.5/dbus/1.11.2                            gcc/4.8.5/libfs/1.0.7                gcc/4.8.5/pcre/8.40
 
    gcc/4.8.5/dos2unix/7.3.4                        gcc/4.8.5/libgcrypt/1.6.2            gcc/4.8.5/perl/5.24.1
 
    gcc/4.8.5/expat/2.2.0                            gcc/4.8.5/libgd/2.2.4                gcc/4.8.5/pixman/0.34.0
 
    gcc/4.8.5/exuberant-ctags/5.8                    gcc/4.8.5/libgpg-error/1.21          gcc/4.8.5/pkg-config/0.29.2
 
    gcc/4.8.5/flex/2.6.3                            gcc/4.8.5/libice/1.0.9              gcc/4.8.5/python/2.7.13
 
    gcc/4.8.5/font-util/1.3.1                        gcc/4.8.5/libiconv/1.15              gcc/4.8.5/python/3.6.0      (D)
 
    gcc/4.8.5/fontcacheproto/0.1.3                  gcc/4.8.5/libogg/1.3.2              gcc/4.8.5/readline/7.0
 
    gcc/4.8.5/fontconfig/2.11.1                      gcc/4.8.5/libpciaccess/0.13.4        gcc/4.8.5/renderproto/0.11.1
 
    gcc/4.8.5/fontsproto/2.1.3                      gcc/4.8.5/libpng/1.6.27              gcc/4.8.5/ruby/2.2.0
 
    gcc/4.8.5/fonttosfnt/1.0.4                      gcc/4.8.5/libpthread-stubs/0.3      gcc/4.8.5/sqlite/3.18.0
 
    gcc/4.8.5/freetype/2.7                          gcc/4.8.5/libsigsegv/2.11            gcc/4.8.5/tar/1.29
 
    gcc/4.8.5/gawk/4.1.4                            gcc/4.8.5/libsm/1.2.2                gcc/4.8.5/tcl/8.6.6
 
    gcc/4.8.5/gdbm/1.13                              gcc/4.8.5/libtiff/4.0.6              gcc/4.8.5/tk/8.6.6
 
    gcc/4.8.5/gettext/0.19.8.1                      gcc/4.8.5/libtool/2.4.6              gcc/4.8.5/unzip/6.0
 
    gcc/4.8.5/ghostscript-fonts/8.11                gcc/4.8.5/libunistring/0.9.7        gcc/4.8.5/util-macros/1.19.1
 
    gcc/4.8.5/ghostscript/9.21                      gcc/4.8.5/libunwind/1.1              gcc/4.8.5/vim/8.0.0503
 
    gcc/4.8.5/git/2.12.1                            gcc/4.8.5/libx11/1.6.3              gcc/4.8.5/wget/1.17
 
    gcc/4.8.5/glib/2.49.7                            gcc/4.8.5/libxau/1.0.8              gcc/4.8.5/xcb-proto/1.12
 
    gcc/4.8.5/glpk/4.61                              gcc/4.8.5/libxcb/1.12                gcc/4.8.5/xextproto/7.3.0
 
    gcc/4.8.5/gmake/4.0                              gcc/4.8.5/libxdmcp/1.1.2             gcc/4.8.5/xproto/7.0.29
 
    gcc/4.8.5/gmp/6.1.2                              gcc/4.8.5/libxext/1.3.3              gcc/4.8.5/xtrans/1.3.5
 
    gcc/4.8.5/gnutls/3.5.10                          gcc/4.8.5/libxml2/2.9.4              gcc/4.8.5/xz/5.2.3
 
    gcc/4.8.5/go-bootstrap/1.4-bootstrap-20161024    gcc/4.8.5/libxrender/0.9.10          gcc/4.8.5/yasm/1.3.0
 
    gcc/4.8.5/go/1.8.1                              gcc/4.8.5/lmod/7.4.11                gcc/4.8.5/zlib/1.2.11
 
    gcc/4.8.5/gperf/3.0.4                            gcc/4.8.5/lua-luafilesystem/1_6_3
 
    gcc/4.8.5/gperftools/2.4                        gcc/4.8.5/lua-luaposix/33.4.0
 
  
 
Please note that this is work in progress and the module names might change. Currently, the number of software packages provided on Leonhard is not comparable to the software we provide on the Euler cluster, but it will grow over time.
 
Please note that this is work in progress and the module names might change. Currently, the number of software packages provided on Leonhard is not comparable to the software we provide on the Euler cluster, but it will grow over time.
  
==Storage==
+
===Hierarchical modules===
Like on the Euler cluster, every user also has a home directory and a personal scratch directory:
+
LMOD allows to define a hierarchy of modules containing 3 layers (Core, Compiler, MPI). The '''core''' layer contains all module files which are not depending on any compiler/MPI. The '''compiler''' layer contains all modules which are depending on a particular compilers, but not on any MPI library. The '''MPI''' layer contains modules that are depending on a particular compiler/MPI combination.
 +
 
 +
When you login to the Leonhard cluster, the standard compiler gcc/4.8.5 is automatically loaded. Running the '''module avail''' command displays all modules that are available for gcc/4.8.5. If you would like to see the modules available for a different compiler, for instance gcc/6.3.0, then you would need to load the compiler module and run '''module avail''' again. For checking out the available modules for gcc/4.8.5 openmpi/2.1.0, you would load the corresponding compiler and MPI module and run again ''module avail'.
 +
 
 +
As a consequence of the module hierachy, you can never have two different versions of the same module loaded at the same time. This helps to avoid problems arising due to misconfiguration of the environment.
  
/cluster/home/''username''
 
/cluster/scratch/''username''
 
  
==Submitting jobs==
 
  
Leonhard uses the same LSF batch system as the [[Euler|Euler cluster]].
+
==Batch system==
 +
Leonhard uses the same LSF batch system as the [[Euler|Euler cluster]]. You can find some general information about the batch system in the [[Using_the_batch_system|Using the batch system]] tutorial.
  
Use the “bsub” command to submit a job and specify resources needed to run your job. By default, a job will get 1&nbsp;core and 1024&nbsp;MB of RAM for 4&nbsp;hours. Unless otherwise specified, jobs requesting more than 36&nbsp;cores will run on a single node. Regular nodes have 36&nbsp;cores and 128 or 512&nbsp;GB of RAM (of which about 90 and 460&nbsp;GB, respectively, are usable).
+
Unless otherwise specified, jobs requesting more than 36&nbsp;cores will run on a single node. Regular nodes have 36&nbsp;cores and 128 or 512&nbsp;GB of RAM (of which about 90 and 460&nbsp;GB, respectively, are usable).
  
Unlike Euler, '''requested memory is strictly enforced as a memory limit.''' For example, if you do not specifically state a memory requirement, your program can not use more than 1&nbsp;GB of RAM per core. What counts is is actually used memory, including page cache for your job. All processes from the same job on a node share the same pool. For example, with a job submitted as
+
Unlike Euler, '''requested memory is strictly enforced as a memory limit.'''  
bsub -n 16 -R "rusage[mem=1024] span[ptile=8]" mpirun ./my_job
 
all of the 8&nbsp;MPI ranks on a single node can use up to 8&nbsp;GB.
 
  
 
===Submitting GPU jobs===
 
===Submitting GPU jobs===
Line 84: Line 74:
 
All GPUs in Leonhard are configured in Exclusive Process mode. The GPU nodes have 20&nbsp;cores, 8&nbsp;GPUs, and 256&nbsp;GB of RAM (of which only about 210&nbsp;GB is usable). To run multi-node job, you will need to request <tt>span[ptile=20]</tt>.
 
All GPUs in Leonhard are configured in Exclusive Process mode. The GPU nodes have 20&nbsp;cores, 8&nbsp;GPUs, and 256&nbsp;GB of RAM (of which only about 210&nbsp;GB is usable). To run multi-node job, you will need to request <tt>span[ptile=20]</tt>.
  
The LSF batch system has partial integrated support for GPUs. To use the GPUs for a job node you need to request the '''ngpus_excl_p''' resource. It refers to the number of GPUs '''per node''', which is unlike other resources, which are requested '''per core'''.
+
The LSF batch system has partial integrated support for GPUs. To use the GPUs for a job node you need to request the '''ngpus_excl_p''' resource. It refers to the number of GPUs '''per node'''. This is unlike other resources, which are requested '''per core'''.
  
 
For example, to run a serial job with one GPU,
 
For example, to run a serial job with one GPU,
Line 94: Line 84:
  
 
While your jobs will see all GPUs, LSF will set the [https://devblogs.nvidia.com/parallelforall/cuda-pro-tip-control-gpu-visibility-cuda_visible_devices/ CUDA_VISIBLE_DEVICES] environment variable, which is honored by CUDA programs.
 
While your jobs will see all GPUs, LSF will set the [https://devblogs.nvidia.com/parallelforall/cuda-pro-tip-control-gpu-visibility-cuda_visible_devices/ CUDA_VISIBLE_DEVICES] environment variable, which is honored by CUDA programs.
 +
 +
===TensorFlow example===
 +
As an example for running a TensorFlow job on a GPU node, we are printing out the TensorFlow version, the string '''Hello TensorFlow!''' and the result of a simple matrix multiplication:
 +
 
 +
[leonhard@lo-login-01 ~]$ '''cd testrun/python'''
 +
[leonhard@lo-login-01 python]$ '''module load python_gpu/2.7.13'''
 +
[leonhard@lo-login-01 python]$ '''cat tftest1.py'''
 +
#/usr/bin/env python
 +
from __future__ import print_function
 +
import tensorflow as tf
 +
 +
vers = tf.__version__
 +
print(vers)
 +
hello = tf.constant('Hello, TensorFlow!')
 +
<nowiki>matrix1 = tf.constant([[3., 3.]])</nowiki>
 +
matrix2 = tf.constant([[2.],[2.]])
 +
product = tf.matmul(matrix1, matrix2)
 +
 +
sess = tf.Session()
 +
print(sess.run(hello))
 +
print(sess.run(product))
 +
sess.close()
 +
[leonhard@lo-login-01 python]$ '''bsub -n 1 -W 4:00 -R "rusage[mem=2048, ngpus_excl_p=1]" python tftest1.py'''
 +
Generic job.
 +
Job <10620> is submitted to queue <gpu.4h>.
 +
[leonhard@lo-login-01 python]$ '''bjobs'''
 +
JOBID      USER      STAT  QUEUE      FROM_HOST  EXEC_HOST  JOB_NAME  SUBMIT_TIME
 +
10620      leonhard  PEND  gpu.4h    lo-login-01            *tftest.py Sep 28 08:02
 +
[leonhard@lo-login-01 python]$ '''bjobs'''
 +
JOBID      USER      STAT  QUEUE      FROM_HOST  EXEC_HOST  JOB_NAME  SUBMIT_TIME
 +
10620      leonhard  RUN  gpu.4h    lo-login-01 lo-gtx-001  *ftest1.py Sep 28 08:03
 +
[leonhard@lo-login-01 python]$ '''bjobs'''
 +
No unfinished job found
 +
[leonhard@lo-login-01 python]$ '''grep -A3 "Creating TensorFlow device" lsf.o10620'''
 +
2017-09-28 08:08:43.235886: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1045] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX 1080, pci bus id: 0000:04:00.0)
 +
1.3.0
 +
Hello, TensorFlow!
 +
[[ 12.]]
 +
[leonhard@lo-login-01 python]$
 +
 +
Please note, that your job will crash if you are running the GPU version of TensorFlow on a CPU node, because TensorFlow is checking on start up if the compute node has a GPU driver.
 +
 +
==Third-party applications==
 +
 +
===Python on Leonhard===
 +
Because certain Python packages need different installations for their CPU and GPU versions, we decided to have separate Python installations with regards to using CPUs and GPUs.
 +
 +
{|class="wikitable" border=1 style="width: 65%;"
 +
! CPU version !! GPU version
 +
|-
 +
|module load python_cpu/3.6.1 || module load python_gpu/3.6.1
 +
|}
 +
 +
===TensorFlow===
 +
On Leonhard, we provide several versions of TensorFlow. The following combinations are available:
 +
 +
{|class="wikitable" border=1 style="width: 65%;"
 +
|-
 +
! colspan=2 | CPU
 +
|-
 +
! Module command !! TensorFlow version
 +
|-
 +
|module load python_cpu/2.7.12
 +
|Python 2.7.12, TensorFlow 1.2.1
 +
|-
 +
|module load python_cpu/2.7.13
 +
|Python 2.7.13, TensorFlow 1.3
 +
|-
 +
|module load python_cpu/2.7.14
 +
|Python 2.7.14, TensorFlow 1.7
 +
|-
 +
|module load python_cpu/3.6.0
 +
|Python 3.6.0, TensorFlow 1.2.1
 +
|-
 +
|module load python_cpu/3.6.1
 +
|Python 3.6.1, TensorFlow 1.3
 +
|-
 +
|module load python_cpu/3.6.4
 +
|Python 3.6.4, TensorFlow 1.7
 +
|-
 +
! colspan=2 | GPU
 +
|-
 +
! Module command !! TensorFlow version
 +
|-
 +
|module load python_gpu/2.7.12
 +
|Python 2.7.12, TensorFlow 1.2.1, CUDA 8.0.61, cuDNN 5.1
 +
|-
 +
|module load python_gpu/2.7.13
 +
|Python 2.7.13, TensorFlow 1.3, CUDA 8.0.61, cuDNN 6.0
 +
|-
 +
|module load python_gpu/2.7.14
 +
|Python 2.7.14, TensorFlow 1.7, CUDA 9.0.176, cuDNN 7.0
 +
|-
 +
|module load python_gpu/3.6.0
 +
|Python 3.6.0, TensorFlow 1.2.1, CUDA 8.0.61, cuDNN 5.1
 +
|-
 +
|module load python_gpu/3.6.1
 +
|Python 3.6.1, TensorFlow 1.3, CUDA 8.0.61, cuDNN 6.0
 +
|-
 +
|module load python_gpu/3.6.4
 +
|Python 3.6.4, TensorFlow 1.7, CUDA 9.0.176, cuDNN 7.0
 +
|}
 +
 +
If you would like to run a TensorFlow job on a CPU node, then you would need to load a CPU version of TensorFlow, whereas you would need to load a GPU version of TensorFlow for running a TensorFlow job on a GPU node.
 +
 +
==Troubleshooting==
 +
 +
===I can no longer access the Leonhard Open cluster===
 +
Some users only got temporary access to the Leonhard Open cluster for a course or a project, which is limited in time. The HPC group does not manage membership of the shareholder groups on Leonhard. The shareholder groups are defined through a custom ETH group. If you could access Leonhard Open for some time and can no longer access it, then you were most likely removed from the custom ETH group, which was used to define the share. If this is the case, then please contact the local IT support group (ISG) of your department for further information, as they are managing the custom ETH groups.
 +
 +
===I get an error message, when running my software that uses a GPU===
 +
If you are getting error messages, about not finding a cuda library, like for instance:
 +
 +
ImportError: libcuda.so.1: cannot open shared object file: No such file or directory
 +
 +
or
 +
 +
ImportError: libcublas.so.9.0: cannot open shared object file: No such file or directory
 +
 +
Then you are most likely running the software on a login node or a compute node without GPU's.
 +
 +
If you would like to run a software that requires access to the GPU driver, then you need to submit it as a batch job and [[Leonhard_beta_testing#Submitting_GPU_jobs | request a GPU]] from the batch system.
 +
 +
===Cluster is missing the h5py python package===
 +
The h5py Python package is linked against the HDF5 library, therefore you need to also load the HDF5 module, such that h5py can located the HDF5 libraries.
 +
 +
[leonhard@lo-s4-019 ~]$ '''module load python_gpu/3.6.1 hdf5/1.10.1'''
 +
[leonhard@lo-s4-019 ~]$ '''python'''
 +
Python 3.6.1 (default, Sep 27 2017, 13:27:13)
 +
[GCC 4.8.5 20150623 (Red Hat 4.8.5-11)] on linux
 +
Type "help", "copyright", "credits" or "license" for more information.
 +
>>> '''import h5py'''
 +
>>> '''h5py.__version__'''
 +
'2.7.1'
 +
>>>

Latest revision as of 08:42, 16 September 2021

This page contains information about the Leonhard Open cluster, which is now obsolete as the cluster has been integrated into the Euler cluster on 14/15 September 2021

Please read through the following to get started on the Leonhard cluster.

Current status

The Leonhard Open cluster is still in the beta testing phase.

The lo-a2-*, the lo-gtx-* and the lo-s4-* compute nodes are connected to the InfiniBand high-performance interconnect and can access the GPFS storage system (/cluster/scratch,/cluster/work) and the NetApp storage system (/cluster/project,/cluster/home).

Accessing the cluster

Who can access the cluster

Access is restricted to Leonhard shareholders and groups that want to test it before investing. Guest users cannot access the Leonhard cluster.

SSH

You can find general information about how to access our clusters via SSH in the Accessing the clusters tutorial.

The command to access the Leonhard cluster via SSH is:

ssh username@login.leonhard.ethz.ch

where username corresponds to your ETH username.

Storage

You can find general information about the storage systems in the Storage systems tutorial.

Like on the Euler cluster, every user also has a home directory and a personal scratch directory:

/cluster/home/username
/cluster/scratch/username

Modules for preparing your environment

LMOD

For the Leonhard cluster, we decided to switch from the environment modules that are used on the Euler cluster to Lmod modules, which provide some additional features. You should barely notice the transition from environment modules to Lmod modules as the commands are mostly the same. Therefore please refer to the Setting up your environment tutorial for a general documentation about the module commands.

[leonhard@lo-login-02 ~]$ module avail boost

----------------------------------------- /cluster/apps/lmodules/Compiler/gcc/4.8.5 ------------------------------------------
   boost/1.63.0

Use "module spider" to find all possible modules.
Use "module keyword key1 key2 ..." to search for all possible modules matching any of the "keys".


[leonhard@lo-login-02 ~]$ module load boost/1.63.0
[leonhard@lo-login-02 ~]$ module list

Currently Loaded Modules:
  1) gcc/4.8.5   2) StdEnv   3) boost/1.63.0

[leonhard@lo-login-02 ~]$ 

Please note that this is work in progress and the module names might change. Currently, the number of software packages provided on Leonhard is not comparable to the software we provide on the Euler cluster, but it will grow over time.

Hierarchical modules

LMOD allows to define a hierarchy of modules containing 3 layers (Core, Compiler, MPI). The core layer contains all module files which are not depending on any compiler/MPI. The compiler layer contains all modules which are depending on a particular compilers, but not on any MPI library. The MPI layer contains modules that are depending on a particular compiler/MPI combination.

When you login to the Leonhard cluster, the standard compiler gcc/4.8.5 is automatically loaded. Running the module avail command displays all modules that are available for gcc/4.8.5. If you would like to see the modules available for a different compiler, for instance gcc/6.3.0, then you would need to load the compiler module and run module avail again. For checking out the available modules for gcc/4.8.5 openmpi/2.1.0, you would load the corresponding compiler and MPI module and run again module avail'.

As a consequence of the module hierachy, you can never have two different versions of the same module loaded at the same time. This helps to avoid problems arising due to misconfiguration of the environment.


Batch system

Leonhard uses the same LSF batch system as the Euler cluster. You can find some general information about the batch system in the Using the batch system tutorial.

Unless otherwise specified, jobs requesting more than 36 cores will run on a single node. Regular nodes have 36 cores and 128 or 512 GB of RAM (of which about 90 and 460 GB, respectively, are usable).

Unlike Euler, requested memory is strictly enforced as a memory limit.

Submitting GPU jobs

All GPUs in Leonhard are configured in Exclusive Process mode. The GPU nodes have 20 cores, 8 GPUs, and 256 GB of RAM (of which only about 210 GB is usable). To run multi-node job, you will need to request span[ptile=20].

The LSF batch system has partial integrated support for GPUs. To use the GPUs for a job node you need to request the ngpus_excl_p resource. It refers to the number of GPUs per node. This is unlike other resources, which are requested per core.

For example, to run a serial job with one GPU,

bsub -R "rusage[ngpus_excl_p=1]" ./my_cuda_program

or on a full node with all eight GPUs and up to 90 GB of RAM,

bsub -n 20 -R "rusage[mem=4500,ngpus_excl_p=8]" ./my_cuda_program

or on two full nodes:

bsub -n 40 -R "rusage[mem=4500,ngpus_excl_p=8] span[ptile=20]" ./my_cuda_program

While your jobs will see all GPUs, LSF will set the CUDA_VISIBLE_DEVICES environment variable, which is honored by CUDA programs.

TensorFlow example

As an example for running a TensorFlow job on a GPU node, we are printing out the TensorFlow version, the string Hello TensorFlow! and the result of a simple matrix multiplication:

[leonhard@lo-login-01 ~]$ cd testrun/python
[leonhard@lo-login-01 python]$ module load python_gpu/2.7.13
[leonhard@lo-login-01 python]$ cat tftest1.py
#/usr/bin/env python
from __future__ import print_function
import tensorflow as tf

vers = tf.__version__
print(vers)
hello = tf.constant('Hello, TensorFlow!')
matrix1 = tf.constant([[3., 3.]])
matrix2 = tf.constant([[2.],[2.]])
product = tf.matmul(matrix1, matrix2)

sess = tf.Session()
print(sess.run(hello))
print(sess.run(product))
sess.close()
[leonhard@lo-login-01 python]$ bsub -n 1 -W 4:00 -R "rusage[mem=2048, ngpus_excl_p=1]" python tftest1.py
Generic job.
Job <10620> is submitted to queue <gpu.4h>.
[leonhard@lo-login-01 python]$ bjobs
JOBID      USER      STAT  QUEUE      FROM_HOST   EXEC_HOST   JOB_NAME   SUBMIT_TIME
10620      leonhard  PEND  gpu.4h     lo-login-01             *tftest.py Sep 28 08:02
[leonhard@lo-login-01 python]$ bjobs
JOBID      USER      STAT  QUEUE      FROM_HOST   EXEC_HOST   JOB_NAME   SUBMIT_TIME
10620      leonhard  RUN   gpu.4h     lo-login-01 lo-gtx-001  *ftest1.py Sep 28 08:03
[leonhard@lo-login-01 python]$ bjobs
No unfinished job found
[leonhard@lo-login-01 python]$ grep -A3 "Creating TensorFlow device" lsf.o10620
2017-09-28 08:08:43.235886: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1045] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX 1080, pci bus id: 0000:04:00.0)
1.3.0
Hello, TensorFlow!
 12.
[leonhard@lo-login-01 python]$

Please note, that your job will crash if you are running the GPU version of TensorFlow on a CPU node, because TensorFlow is checking on start up if the compute node has a GPU driver.

Third-party applications

Python on Leonhard

Because certain Python packages need different installations for their CPU and GPU versions, we decided to have separate Python installations with regards to using CPUs and GPUs.

CPU version GPU version
module load python_cpu/3.6.1 module load python_gpu/3.6.1

TensorFlow

On Leonhard, we provide several versions of TensorFlow. The following combinations are available:

CPU
Module command TensorFlow version
module load python_cpu/2.7.12 Python 2.7.12, TensorFlow 1.2.1
module load python_cpu/2.7.13 Python 2.7.13, TensorFlow 1.3
module load python_cpu/2.7.14 Python 2.7.14, TensorFlow 1.7
module load python_cpu/3.6.0 Python 3.6.0, TensorFlow 1.2.1
module load python_cpu/3.6.1 Python 3.6.1, TensorFlow 1.3
module load python_cpu/3.6.4 Python 3.6.4, TensorFlow 1.7
GPU
Module command TensorFlow version
module load python_gpu/2.7.12 Python 2.7.12, TensorFlow 1.2.1, CUDA 8.0.61, cuDNN 5.1
module load python_gpu/2.7.13 Python 2.7.13, TensorFlow 1.3, CUDA 8.0.61, cuDNN 6.0
module load python_gpu/2.7.14 Python 2.7.14, TensorFlow 1.7, CUDA 9.0.176, cuDNN 7.0
module load python_gpu/3.6.0 Python 3.6.0, TensorFlow 1.2.1, CUDA 8.0.61, cuDNN 5.1
module load python_gpu/3.6.1 Python 3.6.1, TensorFlow 1.3, CUDA 8.0.61, cuDNN 6.0
module load python_gpu/3.6.4 Python 3.6.4, TensorFlow 1.7, CUDA 9.0.176, cuDNN 7.0

If you would like to run a TensorFlow job on a CPU node, then you would need to load a CPU version of TensorFlow, whereas you would need to load a GPU version of TensorFlow for running a TensorFlow job on a GPU node.

Troubleshooting

I can no longer access the Leonhard Open cluster

Some users only got temporary access to the Leonhard Open cluster for a course or a project, which is limited in time. The HPC group does not manage membership of the shareholder groups on Leonhard. The shareholder groups are defined through a custom ETH group. If you could access Leonhard Open for some time and can no longer access it, then you were most likely removed from the custom ETH group, which was used to define the share. If this is the case, then please contact the local IT support group (ISG) of your department for further information, as they are managing the custom ETH groups.

I get an error message, when running my software that uses a GPU

If you are getting error messages, about not finding a cuda library, like for instance:

ImportError: libcuda.so.1: cannot open shared object file: No such file or directory

or

ImportError: libcublas.so.9.0: cannot open shared object file: No such file or directory

Then you are most likely running the software on a login node or a compute node without GPU's.

If you would like to run a software that requires access to the GPU driver, then you need to submit it as a batch job and request a GPU from the batch system.

Cluster is missing the h5py python package

The h5py Python package is linked against the HDF5 library, therefore you need to also load the HDF5 module, such that h5py can located the HDF5 libraries.

[leonhard@lo-s4-019 ~]$ module load python_gpu/3.6.1 hdf5/1.10.1
[leonhard@lo-s4-019 ~]$ python
Python 3.6.1 (default, Sep 27 2017, 13:27:13)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-11)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import h5py
>>> h5py.__version__
'2.7.1'
>>>