Difference between revisions of "Deploy a container using Singularity"
From ScientificComputing
| Line 4: | Line 4: | ||
* [[Deploy a container using Singularity#Method 1: Use the Docker image on Dockerhub|Method 1: Use the Docker image on Dockerhub]] | * [[Deploy a container using Singularity#Method 1: Use the Docker image on Dockerhub|Method 1: Use the Docker image on Dockerhub]] | ||
* [[Deploy a container using Singularity#Method 2: Convert a Docker image to a Singularity image |Method 2: Convert a Docker image to a Singularity image]] | * [[Deploy a container using Singularity#Method 2: Convert a Docker image to a Singularity image |Method 2: Convert a Docker image to a Singularity image]] | ||
| − | * [[Deploy a container using Singularity#Method 3: Build a container image from Singularity | + | * [[Deploy a container using Singularity#Method 3: Build a container image from Singularity definition file | Method 3: Build a container image from Singularity definition file]] |
| − | + | = Create a Docker file and build a docker image on your local computer = | |
* [https://docs.docker.com/get-docker/ Install Docker] | * [https://docs.docker.com/get-docker/ Install Docker] | ||
* Create a folder | * Create a folder | ||
| Line 43: | Line 43: | ||
dockeruser/ubuntu-python3 latest d639b6f22ee5 18 minutes ago 2.19GB | dockeruser/ubuntu-python3 latest d639b6f22ee5 18 minutes ago 2.19GB | ||
| − | + | = Deploy a container on the cluster = | |
| − | + | == Method 1: Use the Docker image on Dockerhub == | |
* On your local computer, log into your Docker account (replace ''dockeruser'' with your Docker username) | * On your local computer, log into your Docker account (replace ''dockeruser'' with your Docker username) | ||
[user@local]$ docker login | [user@local]$ docker login | ||
| Line 131: | Line 131: | ||
>>> | >>> | ||
| − | + | == Method 2: Convert a Docker image to a Singularity image == | |
* Create a tarball of the Docker image | * Create a tarball of the Docker image | ||
[user@local]$ docker save d639b6f22ee5 -o ubuntu-python3.tar | [user@local]$ docker save d639b6f22ee5 -o ubuntu-python3.tar | ||
| Line 184: | Line 184: | ||
[nethzuser@eu-ms-001-02 ~]$ | [nethzuser@eu-ms-001-02 ~]$ | ||
| + | == Method 3: Build a container image from Singularity definition file == | ||
| + | Similar to Dockerfile for creating a Docker container, Singularity can build a container from a [https://sylabs.io/guides/3.7/user-guide/definition_files.html Singularity definition file] as well. | ||
| + | |||
| + | |||
| + | Currently, Singularity on Euler is '''not''' setup to build a container from a definition file. | ||
{{back_to_tutorials}} | {{back_to_tutorials}} | ||
Revision as of 07:53, 14 June 2021
|
< Examples |
In this tutorial, we show you how to deploy a container on Euler by using Singularity. The three methods that we present here are:
- Method 1: Use the Docker image on Dockerhub
- Method 2: Convert a Docker image to a Singularity image
- Method 3: Build a container image from Singularity definition file
Create a Docker file and build a docker image on your local computer
- Install Docker
- Create a folder
[user@local]$ mkdir test_container [user@local]$ cd test_container
- Open a file named Dockerfile with a text editor and add the following lines into the Dockerfile to create a container with Ubuntu as an OS and install Python3 with Numpy, Matplotlib and TensorFlow packages
FROM ubuntu:latest WORKDIR /home/project RUN apt-get -y update RUN apt-get install -y lsb-release python3 python3-pip \ && pip install numpy==1.19 matplotlib tensorflow
- Build an image
[user@local]$ docker build -t dockeruser/ubuntu-python3:latest . Sending build context to Docker daemon 3.072kB Step 1/4 : FROM ubuntu:latest ---> 7e0aa2d69a15 Step 2/4 : WORKDIR /home/project ---> Using cache ---> d9947182f6a8 Step 3/4 : RUN apt-get -y update ---> Using cache ---> 89a634ac491b Step 4/4 : RUN apt-get install -y lsb-release python3 python3-pip && pip install numpy==1.19 matplotlib tensorflow ---> Using cache ---> d639b6f22ee5 Successfully built d639b6f22ee5 Successfully tagged dockeruser/ubuntu-python3:latest
[user@local]$ docker images REPOSITORY TAG IMAGE ID CREATED SIZE dockeruser/ubuntu-python3 latest d639b6f22ee5 18 minutes ago 2.19GB
Deploy a container on the cluster
Method 1: Use the Docker image on Dockerhub
- On your local computer, log into your Docker account (replace dockeruser with your Docker username)
[user@local]$ docker login username: dockeruser password:
- Push the image to Dockerhub
[user@local]$ docker push dockeruser/ubuntu-python3:latest The push refers to repository [docker.io/dockeruser/ubuntu-python3] 4648e930d81f: Pushed 2e38cc729d73: Pushed f281ab5d2fac: Pushed 2f140462f3bc: Mounted from library/ubuntu 63c99163f472: Mounted from library/ubuntu ccdbb80308cc: Mounted from library/ubuntu latest: digest: sha256:cd8a34b30aabe432232787d1e93844cd01027c2b235fb88f106297ed26c1f2ca size: 1576
- Log into Euler (replace nethzuser with your NETHZ username)
[user@local]$ ssh nethzuser@euler.ethz.ch
- Request a compute node with Singularity
[nethzuser@eu-login-01 ~]$ bsub -n 1 -R singularity -R light -Is bash Generic job. Job <174809810> is submitted to queue <light.5d>. <<Waiting for dispatch ...>> <<Starting on eu-ms-001-02>> [nethzuser@eu-ms-001-02 ~]$
- Load module eth_proxy to connect to the internet from a compute node
[nethzuser@eu-ms-001-02 ~]$ module load eth_proxy
- Pull the Docker image with Singularity
[nethzuser@eu-ms-001-02 ~]$ cd $SCRATCH [nethzuser@eu-ms-001-02 nethzuser]$ singularity pull docker://dockeruser/ubuntu-python3 INFO: Converting OCI blobs to SIF format INFO: Starting build... Getting image source signatures Copying blob 345e3491a907 done ... INFO: Creating SIF file...
[nethzuser@eu-ms-001-02 nethzuser]$ ll ubuntu-python3_latest.sif
- Run the container as shell
[nethzuser@eu-ms-001-02 nethzuser]$ singularity shell ubuntu-python3_latest.sif Singularity> Singularity> lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 20.04.2 LTS Release: 20.04 Codename: focal
Singularity> python3 Python 3.8.5 (default, May 27 2021, 13:30:53) [GCC 9.3.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import tensorflow as tf 2021-06-09 14:08:24.919069: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /.singularity.d/libs 2021-06-09 14:08:24.919094: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine. >>> tf.__version__ '2.5.0' >>> Singularity> exit exit
As you can see, the container contains GCC 9.3.0 with Python 3.8.5 and TensorFlow 2.5.0. At the time of writing tutorial, the latest version of GCC is 8.2.0 and TensorFlow 2.4.0
[nethzuser@eu-ms-001-02 nethzuser]$ module avail gcc --------------------------- /cluster/apps/lmodules/Core -------------------------------- gcc/4.8.2 gcc/4.8.5 (L) gcc/5.4.0 gcc/6.3.0 gcc/7.3.0 gcc/8.2.0 (D)
[nethzuser@eu-ms-001-02 nethzuser]$ module load gcc/6.3.0 python/3.8.5 [nethzuser@eu-ms-001-02 nethzuser]$ python Python 3.8.5 (default, Oct 6 2020, 10:04:29) [GCC 6.3.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import tensorflow as tf 2021-06-09 14:56:47.778183: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /cluster/apps/gcc-6.3.0/openblas-0.2.20-cot3cawsqf4pkxjwzjexaykbwn2ch3ii/lib:/cluster/apps/nss/gcc-6.3.0/python/3.8.5/x86_64/lib64:/cluster/spack/apps/linux-centos7-x86_64/gcc-4.8.5/gcc-6.3.0-sqhtfh32p5gerbkvi5hih7cfvcpmewvj/lib64:/cluster/spack/apps/linux-centos7-x86_64/gcc-4.8.5/gcc-6.3.0-sqhtfh32p5gerbkvi5hih7cfvcpmewvj/lib:/cluster/apps/lsf/10.1/linux2.6-glibc2.3-x86_64/lib 2021-06-09 14:56:47.778214: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine. >>> tf.__version__ '2.4.0' >>>
Method 2: Convert a Docker image to a Singularity image
- Create a tarball of the Docker image
[user@local]$ docker save d639b6f22ee5 -o ubuntu-python3.tar [user@local]$ ls ubuntu-python3.tar
- Copy the tarball to $SCRATCH on Euler (replace netzuser with your NETHZ username)
[user@local]$ scp ubuntu-python3.tar netzuser@euler.ethz.ch:/cluster/scratch/nethzuser
- Log in to Euler (replace netzuser with your NETHZ username)
[user@local]$ ssh netzuser@euler.ethz.ch [nethzuser@eu-login-01 ~]$ cd $SCRATCH [nethzuser@eu-login-01 nethzuser]$ pwd /cluster/scratch/nethzuser [nethzuser@eu-login-01 nethzuser]$ pwd /cluster/scratch/nethzuser
- Request a compute node with Singularity
[nethzuser@eu-login-01 ~]$ bsub -n 1 -R singularity -R light -Is bash Generic job. Job <174809810> is submitted to queue <light.5d>. <<Waiting for dispatch ...>> <<Starting on eu-ms-001-02>> [nethzuser@eu-ms-001-02 ~]$
- Convert the tarball to a Singularity image. This shall create a sandbox directory called ubuntu-python3 in your working directory.
[nethzuser@eu-ms-001-02 ~]$ singularity build --sandbox ubuntu-python3 docker-archive://ubuntu-python3.tar INFO: Starting build... ... INFO: Creating sandbox directory... INFO: Build complete: ubuntu-python3
- Run the Singularity box
[nethzuser@eu-ms-001-02 ~]$ singularity shell ubuntu-python3 Singularity> Singularity> gcc --version gcc (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0 Copyright (C) 2019 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. Singularity> python3 --version Python 3.8.5 ingularity> python3 -c "import tensorflow as tf; print(tf.__version__)" 2021-06-10 09:12:44.533833: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /.singularity.d/libs 2021-06-10 09:12:44.533867: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine. 2.5.0 Singularity> exit exit [nethzuser@eu-ms-001-02 ~]$
Method 3: Build a container image from Singularity definition file
Similar to Dockerfile for creating a Docker container, Singularity can build a container from a Singularity definition file as well.
Currently, Singularity on Euler is not setup to build a container from a definition file.
|
< Examples |
