Singularity

From ScientificComputing
Jump to: navigation, search

Introduction

Singularity is a container solution created by necessity for scientific and application driven workloads. The containers can be used to package entire scientific workflows, software and libraries, and even data. You have the possibility to use pre-built containers, create your own containers or even directly run Docker containers.

Note: Singularity has been renamed to apptainer, but on Euler, the command is still singularity.

Installed version

3.8.0

Requirements

Using the HPC infrastructure to run singularity containers can be a security risk for the cluster. Therefore the access is restricted. To get access to it, you can run the following command: get-access.

Workflow

You can either download pre-built containers from external sources (for instance container library or Docker hub), or build your own containers on your workstation (creation of containers requires root privileges and can therefore not be done directly on the HPC clusters).

For the development of a workflow within a container, it might be useful to run singularity in a lightweight batch interactive job: 

srun --ntasks=1 --pty bash

Within such an interactive job, you get a shell directly on one of the compute nodes, which allows to run singularity commands interactively. Once the development of the workflow has finished, the containers can also be run in batch mode: 

sbatch --ntasks=1 --mem-per-cpu=2048 --time=1:00:00 --wrap="singularity run hello-world.simg"

Multi Node Computation

If you plan to use a container on multiple nodes with MPI, you can base your workflow on the following example (code hosted here). In your Dockerfile, you will need to build MPI and then copy it to your own container. Don't forget to compile with UCX in order to have good performances. 

 # Build MPI
 FROM ubuntu:22.04 as mpi
 
 ARG OMPI_VERSION=4.1.4
 ENV OMPI_URL="https://download.open-mpi.org/release/open-mpi/v4.1/openmpi-$OMPI_VERSION.tar.bz2"
 ENV OMPI_DIR=/opt/ompi
 # Very important to have ucx
 ENV OMPI_OPTIONS="--enable-shared --enable-static --disable-silent-rules --with-slurm --with-io-romio-flags='--with-file-system=nfs+ufs+lustre --with-elan=no' --enable-mpi-cxx --with-pmi=/slurm --with-ucx"
 
 # We need to copy our own slurm libraries
 COPY slurm /slurm
 
 # Installing required packages...
 RUN apt-get update && \
     apt-get install -y wget git bash gcc gfortran g++ make file bzip2 libucx0 libucx-dev && \
     rm -rf /var/lib/apt/lists/*
 
 # Installing Open MPI
 RUN mkdir -p /tmp/ompi && \
     mkdir -p /opt && \
     cd /tmp/ompi && \
     wget -O openmpi-${OMPI_VERSION}.tar.bz2 ${OMPI_URL} && \
     tar -xjf openmpi-${OMPI_VERSION}.tar.bz2 && \
     cd /tmp/ompi/openmpi-${OMPI_VERSION} && \
     ./configure --prefix=${OMPI_DIR} ${OMPI_OPTIONS} && \
     make install
 
 # Now we can build whatever image we want
 FROM ubuntu:22.04
 
 # Ensure that OpenMPI can be found
 ENV OMPI_DIR=/opt/ompi
 ENV PATH=$PATH:$OMPI_DIR/bin
 ENV LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$OMPI_DIR/lib
 
 # Install required dependencies for MPI
 RUN apt-get update && \
     apt-get install -y make gcc libucx0 libpmi2-0-dev && \
     rm -rf /var/lib/apt/lists/*
   
 # Copy Open MPI
 COPY --from=mpi /opt/ompi $OMPI_DIR
 
 # Copy own files
 COPY makefile mpiBench /data/
 
 # Compile
 RUN cd /data && \
     make

where the data from slurm can be imported scp -r USER@euler.ethz.ch:/cluster/apps/slurm ..

Finally, you can run the container in slurm with the following script. Don't forget to bind the scratch or you could have issues with temporary files. 

 #!/bin/bash
 
 #SBATCH -n 4 -N 2 --ntasks-per-node=2 -C ib
 #SBATCH --time=00:05:00
 #SBATCH --exclusive --contiguous
 
 module load openmpi
 mpirun -np 4 singularity exec --bind /scratch:/scratch docker://registry.euler.hpc.ethz.ch/node-testing/mpi-bench:latest /data/mpiBench


Security

Singularity is using executables that have the setuid bit set, i.e., they are executed with higher privileges than processes without the setuid bit set. This has implications with regards to security. If there is a vulnerability out in the wild, that can use binaries with setuid bit set for privilege escalation, then we will immediately have to disable singularity until there is a patch that can resolve the issue.

Settings

We recommend you set the following environment variables: 

export APPTAINER_CACHEDIR="$SCRATCH/.singularity"
export APPTAINER_TMPDIR="${TMPDIR:-/tmp}"

The first, APPTAINER_CACHEDIR stores singularity images in your $SCRATCH directory. The second, APPTAINER_TMPDIR, uses the local temporary directory to store temporary data when building images.

Support

For our high-performance clusters, we provide the infrastructure to run singularity containers. We provide help with problems regarding how to run your container on either the Euler or the Leonhard cluster, but we cannot provide support for the software installations inside a container. If you encounter a problem regarding the installation of software inside the container, then please contact the developer that created the container.

Troubleshooting

If you receive an error message, that you cannot run singularity, then please check first if you are a member of the singularity user group (ID-HPC-SINGULARITY). Please run the command 

id | grep ID-HPC-SINGULARITY

on a login node. If you get an empty output 

[samapps@eu-login-16-ng ~]$ id | grep ID-HPC-SINGULARITY
[samapps@eu-login-16-ng ~]$

then your user account is not a member of the singularity user group. If you are member of a shareholder group and would like to use singularity, then please use the command get-access to request access to it.

Useful links

Retrieved from "https://scicomp.ethz.ch/w/index.php?title=Singularity&oldid=10250"