Difference between revisions of "Leonhard beta testing"
(→Storage) |
(→Using modules) |
||
| Line 15: | Line 15: | ||
where ''username'' corresponds to your NETHZ username. | where ''username'' corresponds to your NETHZ username. | ||
| − | == | + | ==Applications== |
For the Leonhard cluster, we decided to switch from the [http://modules.sourceforge.net environment modules] that are used on the Euler cluster to [https://lmod.readthedocs.io/en/latest/index.html Lmod modules], which provide some nice features that are not available for environment modules. You should barely notice the transition from environment modules to Lmod modules as the commands are mostly the same: | For the Leonhard cluster, we decided to switch from the [http://modules.sourceforge.net environment modules] that are used on the Euler cluster to [https://lmod.readthedocs.io/en/latest/index.html Lmod modules], which provide some nice features that are not available for environment modules. You should barely notice the transition from environment modules to Lmod modules as the commands are mostly the same: | ||
Revision as of 13:26, 17 May 2017
The Leonhard cluster is available for early-access beta testing.
Please read through the following to get started.
Contents
Accessing the cluster
Who can access the cluster
Access is restricted to Leonhard shareholders and groups that want to test it before investing. Guest users cannot access the Leonhard cluster.
SSH
Users can access the Leonhard cluster via SSH.
ssh username@login.leonhard.ethz.ch
where username corresponds to your NETHZ username.
Applications
For the Leonhard cluster, we decided to switch from the environment modules that are used on the Euler cluster to Lmod modules, which provide some nice features that are not available for environment modules. You should barely notice the transition from environment modules to Lmod modules as the commands are mostly the same:
[leonhard@lo-login-02 ~]$ module avail ------------------------------------------------- /cluster/spack/lmodules ------------------------------------------------- gcc/4.8.5/apr-util/1.5.4 gcc/4.8.5/guile/2.0.11 gcc/4.8.5/lua/5.3.2 gcc/4.8.5/apr/1.5.2 gcc/4.8.5/help2man/1.47.4 gcc/4.8.5/lz4/1.7.5 gcc/4.8.5/autoconf/2.69 gcc/4.8.5/inputproto/2.3.2 gcc/4.8.5/lzma/4.32.7 gcc/4.8.5/automake/1.15 gcc/4.8.5/jpeg/9b gcc/4.8.5/lzo/2.09 gcc/4.8.5/bash/4.4 gcc/4.8.5/kbproto/1.0.7 gcc/4.8.5/m4/1.4.18 gcc/4.8.5/bdw-gc/7.4.4 gcc/4.8.5/lcms/2.8 gcc/4.8.5/mawk/1.3.4 gcc/4.8.5/binutils/2.28 gcc/4.8.5/libarchive/3.2.1 gcc/4.8.5/mpfr/3.1.5 gcc/4.8.5/bison/3.0.4 gcc/4.8.5/libatomic-ops/7.4.4 gcc/4.8.5/nasm/2.11.06 gcc/4.8.5/bzip2/1.0.6 gcc/4.8.5/libctl/3.2.2 gcc/4.8.5/ncurses/6.0 gcc/4.8.5/coreutils/8.26 gcc/4.8.5/libdrm/2.4.70 gcc/4.8.5/nettle/3.2 gcc/4.8.5/cscope/15.8b gcc/4.8.5/libdwarf/20160507 gcc/4.8.5/openblas/0.2.19 gcc/4.8.5/cuda/8.0.61 gcc/4.8.5/libelf/0.8.13 gcc/4.8.5/openssl/1.0.1e gcc/4.8.5/cudnn/6.0 gcc/4.8.5/libffi/3.2.1 gcc/4.8.5/patch/2.7.5 gcc/4.8.5/curl/7.53.1 gcc/4.8.5/libfontenc/1.1.3 gcc/4.8.5/patchelf/0.9 gcc/4.8.5/dbus/1.11.2 gcc/4.8.5/libfs/1.0.7 gcc/4.8.5/pcre/8.40 gcc/4.8.5/dos2unix/7.3.4 gcc/4.8.5/libgcrypt/1.6.2 gcc/4.8.5/perl/5.24.1 gcc/4.8.5/expat/2.2.0 gcc/4.8.5/libgd/2.2.4 gcc/4.8.5/pixman/0.34.0 gcc/4.8.5/exuberant-ctags/5.8 gcc/4.8.5/libgpg-error/1.21 gcc/4.8.5/pkg-config/0.29.2 gcc/4.8.5/flex/2.6.3 gcc/4.8.5/libice/1.0.9 gcc/4.8.5/python/2.7.13 gcc/4.8.5/font-util/1.3.1 gcc/4.8.5/libiconv/1.15 gcc/4.8.5/python/3.6.0 (D) gcc/4.8.5/fontcacheproto/0.1.3 gcc/4.8.5/libogg/1.3.2 gcc/4.8.5/readline/7.0 gcc/4.8.5/fontconfig/2.11.1 gcc/4.8.5/libpciaccess/0.13.4 gcc/4.8.5/renderproto/0.11.1 gcc/4.8.5/fontsproto/2.1.3 gcc/4.8.5/libpng/1.6.27 gcc/4.8.5/ruby/2.2.0 gcc/4.8.5/fonttosfnt/1.0.4 gcc/4.8.5/libpthread-stubs/0.3 gcc/4.8.5/sqlite/3.18.0 gcc/4.8.5/freetype/2.7 gcc/4.8.5/libsigsegv/2.11 gcc/4.8.5/tar/1.29 gcc/4.8.5/gawk/4.1.4 gcc/4.8.5/libsm/1.2.2 gcc/4.8.5/tcl/8.6.6 gcc/4.8.5/gdbm/1.13 gcc/4.8.5/libtiff/4.0.6 gcc/4.8.5/tk/8.6.6 gcc/4.8.5/gettext/0.19.8.1 gcc/4.8.5/libtool/2.4.6 gcc/4.8.5/unzip/6.0 gcc/4.8.5/ghostscript-fonts/8.11 gcc/4.8.5/libunistring/0.9.7 gcc/4.8.5/util-macros/1.19.1 gcc/4.8.5/ghostscript/9.21 gcc/4.8.5/libunwind/1.1 gcc/4.8.5/vim/8.0.0503 gcc/4.8.5/git/2.12.1 gcc/4.8.5/libx11/1.6.3 gcc/4.8.5/wget/1.17 gcc/4.8.5/glib/2.49.7 gcc/4.8.5/libxau/1.0.8 gcc/4.8.5/xcb-proto/1.12 gcc/4.8.5/glpk/4.61 gcc/4.8.5/libxcb/1.12 gcc/4.8.5/xextproto/7.3.0 gcc/4.8.5/gmake/4.0 gcc/4.8.5/libxdmcp/1.1.2 gcc/4.8.5/xproto/7.0.29 gcc/4.8.5/gmp/6.1.2 gcc/4.8.5/libxext/1.3.3 gcc/4.8.5/xtrans/1.3.5 gcc/4.8.5/gnutls/3.5.10 gcc/4.8.5/libxml2/2.9.4 gcc/4.8.5/xz/5.2.3 gcc/4.8.5/go-bootstrap/1.4-bootstrap-20161024 gcc/4.8.5/libxrender/0.9.10 gcc/4.8.5/yasm/1.3.0 gcc/4.8.5/go/1.8.1 gcc/4.8.5/lmod/7.4.11 gcc/4.8.5/zlib/1.2.11 gcc/4.8.5/gperf/3.0.4 gcc/4.8.5/lua-luafilesystem/1_6_3 gcc/4.8.5/gperftools/2.4 gcc/4.8.5/lua-luaposix/33.4.0
Please note that this is work in progress and the module names might change. Currently, the number of software packages provided on Leonhard is not comparable to the software we provide on the Euler cluster, but it will grow over time.
Storage
Like on the Euler cluster, every user also has a home directory and a personal scratch directory:
/cluster/home/username /cluster/scratch/username
Submitting jobs
Leonhard uses the same LSF batch system as the Euler cluster.
Use the “bsub” command to submit a job and specify resources needed to run your job. By default, a job will get 1 core and 1024 MB of RAM for 4 hours. Unless otherwise specified, jobs requesting more than 36 cores will run on a single node. Regular nodes have 36 cores and 128 or 512 GB of RAM (of which about 90 and 460 GB, respectively, are usable).
Unlike Euler, requested memory is strictly enforced as a memory limit. For example, if you do not specifically state a memory requirement, your program can not use more than 1 GB of RAM per core. What counts is is actually used memory, including page cache for your job. All processes from the same job on a node share the same pool. For example, with a job submitted as
bsub -n 16 -R "rusage[mem=1024] span[ptile=8]" mpirun ./my_job
all of the 8 MPI ranks on a single node can use up to 8 GB.
Submitting GPU jobs
All GPUs in Leonhard are configured in Exclusive Process mode. The GPU nodes have 20 cores, 8 GPUs, and 256 GB of RAM (of which only about 210 GB is usable). To run multi-node job, you will need to request span[ptile=20].
The LSF batch system has partial integrated support for GPUs. To use the GPUs for a job node you need to request the ngpus_excl_p resource. It refers to the number of GPUs per node, which is unlike other resources, which are requested per core.
For example, to run a serial job with one GPU,
bsub -R "rusage[ngpus_excl_p=1]" ./my_cuda_program
or on a full node with all eight GPUs and up to 90 GB of RAM,
bsub -n 20 -R "rusage[mem=4500,ngpus_excl_p=8]" ./my_cuda_program
or on two full nodes:
bsub -n 40 -R "rusage[mem=4500,ngpus_excl_p=8] span[ptile=20]" ./my_cuda_program
While your jobs will see all GPUs, LSF will set the CUDA_VISIBLE_DEVICES environment variable, which is honored by CUDA programs.
