Difference between revisions of "Deploy a container using Singularity"

From ScientificComputing
Jump to: navigation, search
 
(22 intermediate revisions by the same user not shown)
Line 1: Line 1:
 +
__NOTOC__
 
{{back_to_tutorials}}
 
{{back_to_tutorials}}
 +
In this tutorial, we show an example for how to develop a Docker image on you local computer and present three methods for how to deploy a container on the cluster by using Singularity:
 +
* [[Deploy a container using Singularity#Method 1: Use the Docker image on Dockerhub|Method 1: Use the Docker image on Dockerhub]]
 +
* [[Deploy a container using Singularity#Method 2: Convert a Docker image to a Singularity image |Method 2: Convert a Docker image to a Singularity image]]
 +
* [[Deploy a container using Singularity#Method 3: Build a container image from Singularity definition file | Method 3: Build a container image from Singularity definition file]]
  
== Create a Docker file and build a docker image on your local computer ==
+
== Develop a docker image on your local computer ==
 
* [https://docs.docker.com/get-docker/ Install Docker]  
 
* [https://docs.docker.com/get-docker/ Install Docker]  
 
* Create a folder  
 
* Create a folder  
Line 7: Line 12:
 
  [user@local]$ cd test_container
 
  [user@local]$ cd test_container
  
* Open file named **Dockerfile** with a text editor  
+
* Open a file named ''Dockerfile'' with a text editor and add the following lines into the Dockerfile to create a container with Ubuntu as an OS and install Python3 with Numpy, Matplotlib and TensorFlow packages
  
 
  FROM ubuntu:latest
 
  FROM ubuntu:latest
Line 18: Line 23:
  
 
* Build an image
 
* Build an image
  [user@local]$ docker build -t username/ubuntu-python3:latest .
+
  [user@local]$ docker build -t dockeruser/ubuntu-python3:latest .
 
  Sending build context to Docker daemon  3.072kB
 
  Sending build context to Docker daemon  3.072kB
 
  Step 1/4 : FROM ubuntu:latest
 
  Step 1/4 : FROM ubuntu:latest
Line 32: Line 37:
 
   ---> d639b6f22ee5
 
   ---> d639b6f22ee5
 
  Successfully built d639b6f22ee5
 
  Successfully built d639b6f22ee5
  Successfully tagged username/ubuntu-python3:latest
+
  Successfully tagged dockeruser/ubuntu-python3:latest
  
 
  [user@local]$ docker images
 
  [user@local]$ docker images
 
  REPOSITORY                  TAG      IMAGE ID      CREATED          SIZE
 
  REPOSITORY                  TAG      IMAGE ID      CREATED          SIZE
  username/ubuntu-python3    latest    d639b6f22ee5  18 minutes ago  2.19GB
+
  dockeruser/ubuntu-python3    latest    d639b6f22ee5  18 minutes ago  2.19GB
 +
 
 +
== Method 1: Use the Docker image on Dockerhub ==
 +
==== On your local computer ====
 +
* Log into your Docker account (replace ''dockeruser'' with your Docker username)
 +
[user@local]$  docker login
 +
username: ''dockeruser''
 +
password:
  
== Deploy the image on the cluster ==
 
=== Method 1: Use the Docker image on Dockerhub ===
 
 
* Push the image to Dockerhub
 
* Push the image to Dockerhub
  [user@local]$  docker push dockeruser/ubuntu-python3:latest
+
  [user@local]$  docker push ''dockeruser''/ubuntu-python3:latest
 
  The push refers to repository [docker.io/dockeruser/ubuntu-python3]
 
  The push refers to repository [docker.io/dockeruser/ubuntu-python3]
 
  4648e930d81f: Pushed  
 
  4648e930d81f: Pushed  
Line 51: Line 61:
 
  latest: digest: sha256:cd8a34b30aabe432232787d1e93844cd01027c2b235fb88f106297ed26c1f2ca size: 1576
 
  latest: digest: sha256:cd8a34b30aabe432232787d1e93844cd01027c2b235fb88f106297ed26c1f2ca size: 1576
  
* Log into Euler
+
* Log into the Euler cluster (replace ''nethzuser'' with your NETHZ username)
  [user@local]$ ssh nethzuser@euler.ethz.ch
+
  [user@local]$ ssh ''nethzuser''@euler.ethz.ch
  
 +
==== On the cluster ====
 
* [[Singularity#Troubleshooting|To use Singularity, your NETHZ username has to be added to ID-HPC-SINGULARITY group]]  
 
* [[Singularity#Troubleshooting|To use Singularity, your NETHZ username has to be added to ID-HPC-SINGULARITY group]]  
  
Line 63: Line 74:
 
  <<Starting on eu-ms-001-02>>
 
  <<Starting on eu-ms-001-02>>
 
  [nethzuser@eu-ms-001-02 ~]$
 
  [nethzuser@eu-ms-001-02 ~]$
 +
 +
* Load module eth_proxy to connect to the internet from a compute node
 +
[nethzuser@eu-ms-001-02 ~]$ module load eth_proxy
  
 
* Pull the Docker image with Singularity
 
* Pull the Docker image with Singularity
 
  [nethzuser@eu-ms-001-02 ~]$ cd $SCRATCH
 
  [nethzuser@eu-ms-001-02 ~]$ cd $SCRATCH
  [nethzuser@eu-ms-001-02 nethzuser]$ singularity pull docker://dockeruser/ubuntu-python3
+
  [nethzuser@eu-ms-001-02 nethzuser]$ singularity pull docker://''dockeruser''/ubuntu-python3
 
  INFO:    Converting OCI blobs to SIF format
 
  INFO:    Converting OCI blobs to SIF format
 
  INFO:    Starting build...
 
  INFO:    Starting build...
Line 74: Line 88:
 
  INFO:    Creating SIF file...
 
  INFO:    Creating SIF file...
  
  [nethzuser@eu-ms-001-02 nethzuser]$ ll
+
  [nethzuser@eu-ms-001-02 nethzuser]$ ls
 
  ubuntu-python3_latest.sif
 
  ubuntu-python3_latest.sif
  
Line 116: Line 130:
 
  >>> tf.__version__
 
  >>> tf.__version__
 
  '2.4.0'
 
  '2.4.0'
  >>>  
+
  >>>
  
=== Method 2: Convert a Docker image to a Singularity image ===
+
== Method 2: Convert a Docker image to a Singularity image ==
 +
==== On your local computer ====
 
* Create a tarball of the Docker image
 
* Create a tarball of the Docker image
 
  [user@local]$ docker save d639b6f22ee5 -o ubuntu-python3.tar
 
  [user@local]$ docker save d639b6f22ee5 -o ubuntu-python3.tar
Line 124: Line 139:
 
  ubuntu-python3.tar
 
  ubuntu-python3.tar
  
* Copy the tarball to $SCRATCH on Euler
+
* Copy the tarball to $SCRATCH on the cluster (replace ''netzuser'' with your NETHZ username)
  [user@local]$ scp ubuntu-python3.tar netzuser@euler.ethz.ch:/cluster/scratch/nethzuser
+
  [user@local]$ scp ubuntu-python3.tar ''netzuser''@euler.ethz.ch:/cluster/scratch/nethzuser
 +
 
 +
* Log in to the Euler cluster (replace ''netzuser'' with your NETHZ username)
 +
[user@local]$ ssh ''netzuser''@euler.ethz.ch
 +
 
 +
==== On the cluster ====
 +
* Request a compute node with Singularity
 +
[nethzuser@eu-login-01 ~]$ bsub -n 1 -R singularity -R light -Is bash
 +
Generic job.
 +
Job <174809810> is submitted to queue <light.5d>.
 +
<<Waiting for dispatch ...>>
 +
<<Starting on eu-ms-001-02>>
 +
[nethzuser@eu-ms-001-02 ~]$
  
* Log in to Euler
+
* Go to $SCRATCH
[user@local]$ ssh netzuser@euler.ethz.ch
+
  [nethzuser@eu-ms-001-02 ~]$ cd $SCRATCH
  [nethzuser@eu-login-01 ~]$ cd $SCRATCH
+
  [nethzuser@eu-ms-001-02 nethzuser]$ pwd
  [nethzuser@eu-login-01 nethzuser]$ pwd
 
/cluster/scratch/nethzuser
 
 
[nethzuser@eu-login-01 nethzuser]$ pwd
 
 
  /cluster/scratch/nethzuser
 
  /cluster/scratch/nethzuser
  
* Convert the tarbal to a Singularity image
+
* Check if the tarball is in $SCRATCH
  [nethzuser@eu-login-01 nethzuser]$ singularity build --sandbox ubuntu-python3 docker-archive://ubuntu-python3.tar
+
[nethzuser@eu-ms-001-02 nethzuser]$ ls ubuntu-python3.tar
 +
ubuntu-python3.tar
 +
 
 +
* Convert the tarball to a Singularity image. This shall create a sandbox directory called ubuntu-python3 in your working directory.
 +
  [nethzuser@eu-ms-001-02 nethzuser]$ singularity build --sandbox ubuntu-python3 docker-archive://ubuntu-python3.tar
 
  INFO:    Starting build...
 
  INFO:    Starting build...
 
  ...
 
  ...
Line 144: Line 171:
  
 
* Run the Singularity box  
 
* Run the Singularity box  
  [nethzuser@eu-login-01 nethzuser]$ singularity shell ubuntu-python3
+
  [nethzuser@eu-ms-001-02 nethzuser]$ singularity shell ubuntu-python3
 
  Singularity>  
 
  Singularity>  
 
  Singularity> gcc --version
 
  Singularity> gcc --version
Line 151: Line 178:
 
  This is free software; see the source for copying conditions.  There is NO
 
  This is free software; see the source for copying conditions.  There is NO
 
  warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
 
  warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
 
+
 
  Singularity> python3 --version
 
  Singularity> python3 --version
 
  Python 3.8.5
 
  Python 3.8.5
Line 160: Line 187:
 
  Singularity> exit
 
  Singularity> exit
 
  exit
 
  exit
 +
[nethzuser@eu-ms-001-02 nethzuser]$
 +
 +
== Method 3: Build a container image from Singularity definition file ==
 +
Similar to Dockerfile for creating a Docker container, Singularity can build a container from a [https://sylabs.io/guides/3.7/user-guide/definition_files.html Singularity definition file] as well.
  
 
+
Currently, Singularity on the cluster is '''not''' setup to build a container from a definition file.
  
 
{{back_to_tutorials}}
 
{{back_to_tutorials}}

Latest revision as of 07:09, 14 June 2021

< Examples

In this tutorial, we show an example for how to develop a Docker image on you local computer and present three methods for how to deploy a container on the cluster by using Singularity:

Develop a docker image on your local computer

[user@local]$ mkdir test_container
[user@local]$ cd test_container
  • Open a file named Dockerfile with a text editor and add the following lines into the Dockerfile to create a container with Ubuntu as an OS and install Python3 with Numpy, Matplotlib and TensorFlow packages
FROM ubuntu:latest

WORKDIR /home/project

RUN apt-get -y update
RUN apt-get install -y lsb-release python3 python3-pip \
&& pip install numpy==1.19 matplotlib tensorflow
  • Build an image
[user@local]$ docker build -t dockeruser/ubuntu-python3:latest .
Sending build context to Docker daemon  3.072kB
Step 1/4 : FROM ubuntu:latest
 ---> 7e0aa2d69a15
Step 2/4 : WORKDIR /home/project
 ---> Using cache
 ---> d9947182f6a8
Step 3/4 : RUN apt-get -y update
 ---> Using cache
 ---> 89a634ac491b
Step 4/4 : RUN apt-get install -y lsb-release python3 python3-pip && pip install numpy==1.19 matplotlib tensorflow
 ---> Using cache
 ---> d639b6f22ee5
Successfully built d639b6f22ee5
Successfully tagged dockeruser/ubuntu-python3:latest
[user@local]$ docker images
REPOSITORY                   TAG       IMAGE ID       CREATED          SIZE
dockeruser/ubuntu-python3     latest    d639b6f22ee5   18 minutes ago   2.19GB

Method 1: Use the Docker image on Dockerhub

On your local computer

  • Log into your Docker account (replace dockeruser with your Docker username)
[user@local]$  docker login
username: dockeruser
password:
  • Push the image to Dockerhub
[user@local]$  docker push dockeruser/ubuntu-python3:latest
The push refers to repository [docker.io/dockeruser/ubuntu-python3]
4648e930d81f: Pushed 
2e38cc729d73: Pushed 
f281ab5d2fac: Pushed 
2f140462f3bc: Mounted from library/ubuntu 
63c99163f472: Mounted from library/ubuntu 
ccdbb80308cc: Mounted from library/ubuntu 
latest: digest: sha256:cd8a34b30aabe432232787d1e93844cd01027c2b235fb88f106297ed26c1f2ca size: 1576
  • Log into the Euler cluster (replace nethzuser with your NETHZ username)
[user@local]$ ssh nethzuser@euler.ethz.ch

On the cluster

  • Request a compute node with Singularity
[nethzuser@eu-login-01 ~]$ bsub -n 1 -R singularity -R light -Is bash
Generic job.
Job <174809810> is submitted to queue <light.5d>.
<<Waiting for dispatch ...>>
<<Starting on eu-ms-001-02>>
[nethzuser@eu-ms-001-02 ~]$
  • Load module eth_proxy to connect to the internet from a compute node
[nethzuser@eu-ms-001-02 ~]$ module load eth_proxy
  • Pull the Docker image with Singularity
[nethzuser@eu-ms-001-02 ~]$ cd $SCRATCH
[nethzuser@eu-ms-001-02 nethzuser]$ singularity pull docker://dockeruser/ubuntu-python3
INFO:    Converting OCI blobs to SIF format
INFO:    Starting build...
Getting image source signatures
Copying blob 345e3491a907 done  
...
INFO:    Creating SIF file...
[nethzuser@eu-ms-001-02 nethzuser]$ ls
ubuntu-python3_latest.sif
  • Run the container as shell
[nethzuser@eu-ms-001-02 nethzuser]$ singularity shell ubuntu-python3_latest.sif
Singularity> 
Singularity> lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 20.04.2 LTS
Release:	20.04
Codename:	focal
Singularity> python3
Python 3.8.5 (default, May 27 2021, 13:30:53) 
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import tensorflow as tf
2021-06-09 14:08:24.919069: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library  'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /.singularity.d/libs
2021-06-09 14:08:24.919094: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not  have a GPU set up on your machine.
>>> tf.__version__
'2.5.0'
>>> 
Singularity> exit
exit

As you can see, the container contains GCC 9.3.0 with Python 3.8.5 and TensorFlow 2.5.0. At the time of writing tutorial, the latest version of GCC is 8.2.0 and TensorFlow 2.4.0

[nethzuser@eu-ms-001-02 nethzuser]$ module avail gcc
--------------------------- /cluster/apps/lmodules/Core --------------------------------
  gcc/4.8.2    gcc/4.8.5 (L)    gcc/5.4.0    gcc/6.3.0    gcc/7.3.0    gcc/8.2.0 (D)
[nethzuser@eu-ms-001-02 nethzuser]$ module load gcc/6.3.0 python/3.8.5
[nethzuser@eu-ms-001-02 nethzuser]$ python
Python 3.8.5 (default, Oct  6 2020, 10:04:29) 
[GCC 6.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import tensorflow as tf
2021-06-09 14:56:47.778183: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library  'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /cluster/apps/gcc-6.3.0/openblas-0.2.20-cot3cawsqf4pkxjwzjexaykbwn2ch3ii/lib:/cluster/apps/nss/gcc-6.3.0/python/3.8.5/x86_64/lib64:/cluster/spack/apps/linux-centos7-x86_64/gcc-4.8.5/gcc-6.3.0-sqhtfh32p5gerbkvi5hih7cfvcpmewvj/lib64:/cluster/spack/apps/linux-centos7-x86_64/gcc-4.8.5/gcc-6.3.0-sqhtfh32p5gerbkvi5hih7cfvcpmewvj/lib:/cluster/apps/lsf/10.1/linux2.6-glibc2.3-x86_64/lib
2021-06-09 14:56:47.778214: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not  have a GPU set up on your machine.
>>> tf.__version__
'2.4.0'
>>>

Method 2: Convert a Docker image to a Singularity image

On your local computer

  • Create a tarball of the Docker image
[user@local]$ docker save d639b6f22ee5 -o ubuntu-python3.tar
[user@local]$ ls
ubuntu-python3.tar
  • Copy the tarball to $SCRATCH on the cluster (replace netzuser with your NETHZ username)
[user@local]$ scp ubuntu-python3.tar netzuser@euler.ethz.ch:/cluster/scratch/nethzuser
  • Log in to the Euler cluster (replace netzuser with your NETHZ username)
[user@local]$ ssh netzuser@euler.ethz.ch

On the cluster

  • Request a compute node with Singularity
[nethzuser@eu-login-01 ~]$ bsub -n 1 -R singularity -R light -Is bash
Generic job.
Job <174809810> is submitted to queue <light.5d>.
<<Waiting for dispatch ...>>
<<Starting on eu-ms-001-02>>
[nethzuser@eu-ms-001-02 ~]$
  • Go to $SCRATCH
[nethzuser@eu-ms-001-02 ~]$ cd $SCRATCH
[nethzuser@eu-ms-001-02 nethzuser]$ pwd
/cluster/scratch/nethzuser
  • Check if the tarball is in $SCRATCH
[nethzuser@eu-ms-001-02 nethzuser]$ ls ubuntu-python3.tar
ubuntu-python3.tar
  • Convert the tarball to a Singularity image. This shall create a sandbox directory called ubuntu-python3 in your working directory.
[nethzuser@eu-ms-001-02 nethzuser]$ singularity build --sandbox ubuntu-python3 docker-archive://ubuntu-python3.tar
INFO:    Starting build...
...
INFO:    Creating sandbox directory...
INFO:    Build complete: ubuntu-python3
  • Run the Singularity box
[nethzuser@eu-ms-001-02 nethzuser]$ singularity shell ubuntu-python3
Singularity> 
Singularity> gcc --version
gcc (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
Copyright (C) 2019 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Singularity> python3 --version
Python 3.8.5
ingularity> python3 -c "import tensorflow as tf; print(tf.__version__)"
2021-06-10 09:12:44.533833: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /.singularity.d/libs
2021-06-10 09:12:44.533867: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
2.5.0
Singularity> exit
exit
[nethzuser@eu-ms-001-02 nethzuser]$

Method 3: Build a container image from Singularity definition file

Similar to Dockerfile for creating a Docker container, Singularity can build a container from a Singularity definition file as well.

Currently, Singularity on the cluster is not setup to build a container from a definition file.

< Examples