MPI on Euler

From ScientificComputing
Revision as of 07:04, 18 March 2022 by Sfux (talk | contribs) (Intel oneAPI 2022.1.2 [work in progress])

Jump to: navigation, search


The Euler VI and VII nodes have very new Mellanox ConnectX-6 network cards, which provide the Infiniband high-speed interconnect. That jobs can make use of this high-speed interconnect between the compute nodes, the MPI implementation which is used needs to be compatible with the current network driver (MOFED) and may some configuration options need to be set.


OpenMPI 4.0.2 or newer

Jobs using OpenMPI 4.0.2 or new should not have any problems using Infiniband as OpenMPI is compiled with UCX support. No additional configuration is required

OpenMPI older than 4.0.2

OpenMPI versions older than 4.0.2 cannot use the Infiniband network (unless you compiled a version yourself with UCX support) in Euler. The centrally provided installations are using the BTL OpenIB transport layer from verbs, which is no longer supported by the current network driver (MOFED). You can still run jobs with those installations (for instance to reproduce older results), but you would need to disable the BTL OpenIB transport layer with the following mpirun option:

-mca btl ^openib

This will disable the Infiniband high-speed interconnect and jobs will fall back to use the slower ethernet, but this still allows to reproduce older results just with a lower performance.

Intel MPI

Intel MPI requires some configuration options to use the Mellanox infiniband cards, which are different for each version. Users don't know to which node type their jobs get dispatched when not targeting a particular CPU model and therefore we recommend to use a script instead of directly specifying the mpirun command on the bsub command line. In the script you can set configuration options based on the node type. Please find below an example script:


# setting configuration options based on node type the job is running on

# get hosttype
hname=`hostname | cut -d "-" -f 2`

case $hname in
                # euler III node
                # euler IV node
                # euler V node
                # euler VI node
                # euler VII node
                # unknown node type
# command to run
mpirun ...

Intel oneAPI 2022.1.2 [work in progress]

We are still testing configurations to find the most suitable one. There is a new provider called MLX, which should be used.

Intel 19.1.0/2020.1

For intel 19.1.0 (2020.1), please use the following variables

export FI_PROVIDER=verbs
export I_MPI_FABRICS=shm:ofi

In this version the older DAPL provider which can be used with Intel 18.0.1 has been deprecated and can no longer be used.

Intel 18.0.1/2018.1

For Intel 18.0.1 (2018.1) there are two configurations possible.

Configuration 1 (using the OFA provider):

export I_MPI_FABRICS=shm:ofa
export I_MPI_OFA_ADAPTER_NAME="mlx5_0"

Configuration 2 (using the DAPL provider):

export I_MPI_FABRICS=shm:dapl
export I_MPI_DAPL_UD=1
export I_MPI_DAPL_UD_PROVIDER=ofa-v2-mlx5_0-1u

Versions older than 18.0.1

We have not tested the Intel MPI implementations from Intel versions older than 18.0.1. Those installations are from the old software stack an we can no longer support them. They are provided on an as-is basis.