Skip to content

Apptainer

Introduction

Apptainer is a container solution created by necessity for scientific and application driven workloads. The containers can be used to package entire scientific workflows, software and libraries, and even data. You have the possibility to use pre-built containers, create your own containers or even directly run Docker containers.

Note: Singularity has been renamed to Apptainer, but on Euler still both commands singularity and apptainer work.

Installed version

Apptainer 1.3.4

Requirements

Using the HPC infrastructure to run apptainer containers can be a security risk for the cluster. Therefore the access is restricted. To get access to it, you can run the following command: get-access.

Workflow

You can either download pre-built containers from external sources (for instance container library or Docker hub), or build your own containers on your workstation (creation of containers requires root privileges and can therefore not be done directly on the HPC clusters).

For the development of a workflow within a container, it might be useful to run apptainer in a lightweight batch interactive job:

srun --ntasks=1 --pty bash

Within such an interactive job, you get a shell directly on one of the compute nodes, which allows to run apptainer commands interactively. Once the development of the workflow has finished, the containers can also be run in batch mode:

sbatch --ntasks=1 --mem-per-cpu=2048 --time=1:00:00 --wrap="apptainer run hello-world.simg"

Multi Node Computation

If you plan to use a container on multiple nodes with MPI, you can base your workflow on the following example (code hosted here). In your Dockerfile, you will need to build MPI and then copy it to your own container. Don't forget to compile with UCX in order to have good performances.

# Build MPI
FROM ubuntu:22.04 as mpi

ARG OMPI_VERSION=4.1.4
ENV OMPI_URL="https://download.open-mpi.org/release/open-mpi/v4.1/openmpi-$OMPI_VERSION.tar.bz2"
ENV OMPI_DIR=/opt/ompi
# Very important to have ucx
ENV OMPI_OPTIONS="--enable-shared --enable-static --disable-silent-rules --with-slurm --with-io-romio-flags='--with-file-system=nfs+ufs+lustre --with-elan=no' --enable-mpi-cxx --with-pmi=/slurm --with-ucx"

# We need to copy our own slurm libraries
COPY slurm /slurm

# Installing required packages...
RUN apt-get update && \
apt-get install -y wget git bash gcc gfortran g++ make file bzip2 libucx0 libucx-dev && \
rm -rf /var/lib/apt/lists/*

# Installing Open MPI
RUN mkdir -p /tmp/ompi && \
mkdir -p /opt && \
cd /tmp/ompi && \
wget -O openmpi-${OMPI_VERSION}.tar.bz2 ${OMPI_URL} && \
tar -xjf openmpi-${OMPI_VERSION}.tar.bz2 && \
cd /tmp/ompi/openmpi-${OMPI_VERSION} && \
./configure --prefix=${OMPI_DIR} ${OMPI_OPTIONS} && \
make install

# Now we can build whatever image we want
FROM ubuntu:22.04

# Ensure that OpenMPI can be found
ENV OMPI_DIR=/opt/ompi
ENV PATH=$PATH:$OMPI_DIR/bin
ENV LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$OMPI_DIR/lib

# Install required dependencies for MPI
RUN apt-get update && \
apt-get install -y make gcc libucx0 libpmi2-0-dev && \
rm -rf /var/lib/apt/lists/*

# Copy Open MPI
COPY --from=mpi /opt/ompi $OMPI_DIR

# Copy own files
COPY makefile mpiBench /data/

# Compile
RUN cd /data && \
make

where the data from slurm can be imported scp -r USER@euler.ethz.ch:/cluster/apps/slurm ..

Finally, you can run the container in slurm with the following script. Don't forget to bind the scratch or you could have issues with temporary files.

#!/bin/bash

#SBATCH -n 4 -N 2 --ntasks-per-node=2 -C ib
#SBATCH --time=00:05:00
#SBATCH --exclusive --contiguous

module load openmpi
mpirun -np 4 apptainer exec --bind /scratch:/scratch docker://registry.euler.hpc.ethz.ch/node-testing/mpi-bench:latest /data/mpiBench

Security

Apptainer is using executables that have the setuid bit set, i.e., they are executed with higher privileges than processes without the setuid bit set. This has implications with regards to security. If there is a vulnerability out in the wild, that can use binaries with setuid bit set for privilege escalation, then we will immediately have to disable apptainer until there is a patch that can resolve the issue.

Settings

We recommend you set the following environment variables:

export APPTAINER_CACHEDIR="$SCRATCH/.apptainer"
export APPTAINER_TMPDIR="${TMPDIR:-/tmp}"

The first, APPTAINER_CACHEDIR stores apptainer images in your $SCRATCH directory. The second, APPTAINER_TMPDIR, uses the local temporary directory to store temporary data when building images.

Support

For our high-performance clusters, we provide the infrastructure to run apptainer containers. We provide help with problems regarding how to run your container on Euler, but we cannot provide support for the software installations inside a container. If you encounter a problem regarding the installation of software inside the container, then please contact the developer that created the container.

Troubleshooting

If you receive an error message, that you cannot run apptainer, then please check first if you are a member of the apptainer user group (ID-HPC-SINGULARITY). Please run the command

id | grep ID-HPC-SINGULARITY

on a login node. If you get an empty output

[sfux@eu-login-16-ng ~]$ id | grep ID-HPC-SINGULARITY
[sfux@eu-login-16-ng ~]$

then your user account is not a member of the apptainer user group. If you are member of a shareholder group and would like to use apptainer, then please use the command get-access to request access to it.