Software Containers with Singularity

class: center, top, title-slide

# Software Containers with Singularity

### Center for Advanced Research Computing <br> University of Southern California <br>

#### Last updated on 2024-07-22

---

## Outline

1. Overview of software containers
2. Downloading pre-built container images
3. Running containers
4. Building custom container images
5. Inspecting container images

---

## First some terminology

- **Container image** — executable file saved to storage (immutable blueprint)
- **Container** — running instance of a container image
- **Container engine** — application that builds and runs container images
- **Host system** — operating system that runs the container engine
- **Container registry** — catalog of container images (download/upload)

---

## What is a software container?

- A software environment that bundles one or more applications and all their dependencies
- OS-level virtualization with process-level isolation on host system
- Provides a custom user space separate from host system
- Different from a virtual machine (which virtualizes hardware)
  - Containers have direct access to the host kernel
  - Virtual machines have indirect access (performance penalty)

General example:

```bash
Container = Operating system + Libraries + Applications
```

Specific example:

```bash
Container for Python = Debian + libcurl,libpng,etc. + Python + Python packages
```

---

## Container architecture

Source: [Singularity Community and SingularityPRO on high-performance servers](https://sylabs.io/download/singularity-community-and-singularitypro-on-high-performance-servers/)

---

## What do containers enable?

- A software environment that is:
  - Shareable
  - Portable
  - Stable
  - Reproducible
  - Isolated
  - Secure

---

## Diverse container ecosystem

- Container image formats
  - Docker, LXD, etc.
  - Singularity
- Container engines
  - Docker, LXC, etc.
  - Singularity
- Container registries
  - Docker Hub, Quay, etc.
  - Singularity Cloud Library
- Open Container Initiative (OCI) for interoperability
  - Image format standard
  - Runtime standard
  - Distribution standard

---

## Docker vs. Singularity

- Docker
  - Designed for running persistent services (e.g., web apps)
  - Requires superuser privileges to build and run container images
- Singularity
  - Designed for running HPC workloads on shared systems
  - Does not require superuser privileges to run container images
  - But may require superuser privileges to build container images
- Superuser privileges restricted to system admins on shared HPC systems
- Both adhere to OCI image format standard
- Can convert images from Docker to Singularity format
  - Via `singularity pull` or [docker2singularity](https://github.com/singularityhub/docker2singularity)
  - Or vice versa: [singularity2docker](https://github.com/singularityhub/singularity2docker)

---

## Singularity containers

- A container engine designed for HPC systems running Linux
- Originally developed at LBNL
- Spinoff firm [Sylabs](https://sylabs.io/) with open-core model
  - SingularityCE (open-source community edition)
  - SingularityPRO (proprietary edition)
- Can be fully isolated from or integrated with host system
  - Defaults to some integration (e.g., mounting home directory)
  - Useful for parallel file systems, GPUs, high-speed networks
- Alternatives for HPC
  - [Apptainer](https://apptainer.org/)
  - [Sarus](https://sarus.readthedocs.io/en/stable/index.html)
  - [Charliecloud](https://hpc.github.io/charliecloud/)
  - [Shifter](https://shifter.readthedocs.io/en/latest/)
  - [Podman](https://podman.io/)

---

## Why use Singularity for research computing?

- Install anything you want (based on any Linux OS)
- Ease installation issues by using pre-built container images
- Ensure reproducible software environments
- Ensure the same software stack is used among a research group
  - Install once and share
  - Common workflows or pipelines
- Ensure the same software stack is used across Linux systems
  - Workstations
  - HPC centers
  - Cloud computing services
- Use older or development versions of software (not available in software modules)
- Use proprietary software (typically distributed as binaries) that depends on other software not available on host system
- Convert Docker images to Singularity images to run on HPC systems

---

## Limitations of Singularity

- Designed for Linux systems
- Portability depends on a few factors
  - CPU architecture format (x86 vs. ARM)
  - Binary format (ELF)
  - Kernel, glibc, other API compatibility
- Not always backward compatible (v2 vs. v3)
- May need superuser privileges to build images

---

## Singularity Image Format (SIF)

- File format for storing container software environments
- A single immutable, compressed, executable image file (`.sif`)
- If you need to modify an image, you have to rebuild it or use as base image for a new build
- Can adhere to OCI image format standard (with `--oci` option)

---

## Accessing Singularity on CARC clusters

- Access directly
- No software module needed

```bash
module purge

singularity --version
```

---

## How to get container images?

- Pulling (downloading) pre-existing images from container registries
    - [Singularity Cloud Library](https://cloud.sylabs.io/library)
    - [Docker Hub](https://hub.docker.com/)
    - [BioContainers](https://biocontainers.pro)
    - [Dockstore](https://www.dockstore.org/)
    - [NVIDIA GPU Cloud Catalog](https://catalog.ngc.nvidia.com/containers)
- Downloading images from software websites
- Building your own custom image

---

## Pulling existing container images

- Downloading from container registries via `singularity pull`
- Syntax uses registry name, repository name, image name, and tag

First request a compute node:

```bash
salloc -p debug -c 4
```

Some [Docker Hub](https://hub.docker.com/) examples:

```bash
singularity pull docker://python:latest

singularity pull python.sif docker://python:latest

singularity pull docker://python:3.11.9

singularity pull docker://pytorch/pytorch:2.3.1-cuda12.1-cudnn8-runtime
```

---

## Limited /tmp space

- Singularity uses /tmp space when pulling and building images
- For large images, the /tmp space may be too small (limited to 1 GB on CARC nodes)
- Will fail with an error message
- In these cases, use /dev/shm space instead on a compute node (limited by memory request)

First request a compute node with enough memory:

```bash
salloc -p debug -c 8 --mem=32G
```

Change location of temporary files with environment variable:

```bash
mkdir /dev/shm/$USER

export SINGULARITY_TMPDIR=/dev/shm/$USER

singularity pull docker://pytorch/pytorch:2.3.1-cuda12.1-cudnn8-devel
```

---

## Container image cache

- Images are cached when you pull them, which uses storage space
- By default, the cache is located at `~/.singularity/cache`

Change location of cache directory with environment variable:

```bash
mkdir /scratch1/$USER/singularity

export SINGULARITY_CACHEDIR=/scratch1/$USER/singularity
```

List and clean the cache:

```bash
singularity cache list

singularity cache clean
```

---

## Running containers

- Containers can be run in interactive or batch modes
- Various options can be used to isolate/integrate with the host system
- A container process is like any other Linux process

Run an interactive shell within the container:

```bash
singularity shell [options] <image>
```

Run batch commands within the container:

```bash
singularity exec [options] <image> <commands>
```

Run a pre-defined runscript within the container:

```bash
singularity run [options] <image>
```

---

## Examples for running containers

Run an interactive shell within the container:

```bash
singularity shell python.sif
```

Run batch commands within the container:

```bash
singularity exec python.sif python -c 'print("Hello world")'

singularity exec python.sif python script.py
```

Run a pre-defined runscript within the container:

```bash
singularity run python.sif
```

---

## File system within containers

- Unique file system within the container
  - `/bin`
  - `/lib`
  - `/usr`
- Some directories from the host system are automatically mounted to the container
  - `$HOME`
  - `$PWD`
  - `/tmp`
  - `/var/tmp`
  - `/proc`
  - `/sys`
  - `/dev`

---

## Bind mounting directories to containers

- Use `--bind` option to mount files or directories to container
- By default `$HOME` and `$PWD` are mounted
- One or more bind mounts separated by a comma

Add your `/scratch1` directory:

```bash
singularity exec --bind /scratch1/$USER python.sif python script.py
```

Change name when binding if needed:

```bash
singularity exec --bind /scratch1/$USER:/mydir python.sif python script.py
```

Use environment variable instead:

```bash
export SINGULARITY_BIND=/scratch1/$USER,/project/<project_id>

singularity exec python.sif python script.py
```

---

## Useful options for isolation

- Use `--cleanenv` option to minimize environment variables transferred to container
- Use `--no-home` option to exclude `/home1` directory
  - For example, Python or R containers
  - Python and R packages are installed in your home directory by default
  - Can lead to conflicts with software installed in container

```bash
singularity shell --cleanenv --no-home --bind $PWD python.sif

singularity exec --cleanenv --no-home --bind $PWD python.sif python script.py
```

---

## Running containers using GPUs

- Containers need to access host GPU driver
- Use `--nv` option to access NVIDIA GPUs
- Run `nvidia-smi` on NVIDIA GPU node to see current driver version and compatibility
- Use `--rocm` option to access AMD ROCm GPUs (none on CARC clusters)
- [GPU support docs](https://sylabs.io/guides/latest/user-guide/gpu.html)

For example, PyTorch using an NVIDIA GPU:

```bash
singularity shell --nv --no-home --bind $PWD pytorch.sif

singularity exec --nv --no-home --bind $PWD pytorch.sif python script.py
```

---

## Running containers using MPI

- Two approaches: hybrid vs. mount
- Pros and cons for each approach
- Less portable because it depends on MPI versions available on host system
- [MPI support docs](https://sylabs.io/guides/latest/user-guide/mpi.html)

For example, an OpenMPI program:

```bash
srun --mpi=pmix_v2 -n $SLURM_NTASKS singularity exec openmpi.sif ./mpi_program
```

---

## Example Slurm job script

```bash
#!/bin/bash

#SBATCH --account=<project_id>
#SBATCH --partition=main
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=8
#SBATCH --mem=16G
#SBATCH --time=1:00:00

module purge

singularity exec --cleanenv --no-home --bind $PWD python.sif python script.py
```

---

## Example Slurm job script using GPU

```bash
#!/bin/bash

#SBATCH --account=<project_id>
#SBATCH --partition=gpu
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=8
#SBATCH --gpus-per-task=a100:1
#SBATCH --mem=32G
#SBATCH --time=1:00:00

module purge

singularity exec --nv --cleanenv --no-home --bind $PWD pytorch.sif python script.py
```

---

## Building custom container images

- Need to build images outside of CARC clusters, but need a Linux OS
- Can build interactively in sandbox mode (but keep track of commands)
- Will ultimately want to build in batch mode with a definition file (reproducible recipe)
- Best option is to use the cloud-based [Singularity Remote Builder](https://cloud.sylabs.io/builder)
  - Builds in batch mode
  - May run slow on the free tier
- Or use a virtual machine on your local computer (e.g., [Multipass](https://multipass.run/), [Virtual Box](https://www.virtualbox.org/))
  - Install a Linux OS
  - Then install SingularityCE
- Could build container images using Docker, and then convert to SIF
- Could also build container images using Dockerfiles with `--oci` mode
- Could also use Git-based CI/CD as part of build process

---

## Building workflow for CARC clusters

1. Create a definition file
2. Build image externally using the definition file
3. If error occurs, modify definition file and rebuild (back to step 1)
4. Transfer image file to CARC directory
5. Test image
6. If error occurs, modify definition file and rebuild (back to step 1)

---

## Definition files

- Recipe for building a container image
  - Start with a base Linux OS (e.g., Debian, AlmaLinux, etc.)
  - Or start with any existing container image from a registry
  - Then install software, add files, etc.
- Similar to a Dockerfile, but different syntax
- Document and version control definition files for reproducibility
- [CARC Singularity template definition files](https://github.com/uschpc/singularities)

---

## Structure of a definition file

```bash
# Header (required)
Bootstrap: ...
From: ...
# Sections (optional)
%files
    Copy files to the container from host (/source /destination)
%post
    Install software, write configuration files, etc.
%test
    Run commands to validate build process
%environment
    Define environment variables that will be set at runtime
%startscript
    Run commands when singularity instance start command is used
%runscript
    Run commands when singularity run command is used
%labels
    Add metadata labels (name-value pair)
%help
    Describe the container and its intended use
```

---

## Example definition file for Python

```bash
Bootstrap: docker
From: python:latest

%post
    apt-get -y update
    apt-get -y upgrade
    apt-get -y clean

python -m venv /opt/pandas
    . /opt/pandas/bin/activate
    pip install pandas

%test
    . /opt/pandas/bin/activate
    python --version
    pip list

%runscript
    . /opt/pandas/bin/activate
    python

%help
    Debian with latest version of Python and pandas.
```

---

## Using Singularity Remote Builder

- [Singularity Container Services](https://cloud.sylabs.io) by [Sylabs](https://www.sylabs.io)
- Free service with limits
  - 11 GB of storage space
  - 1000 minutes per month of build time
- Log in with other account (GitHub, GitLab, Google, Microsoft)
- Set up [access token](https://cloud.sylabs.io/tokens) for CARC clusters
- Use web interface to build
- Or build remotely from the command line

```bash
singularity remote login

singularity build --remote python.sif python.def
```

---

## Installing optimized software

- "Optimized" typically means targeting specific hardware (e.g., CPU microarchitecture)
- Tradeoff between portability and performance
- Choice of base OS for an image can be important for performance
- Using Linux software package managers (e.g., APT, DNF) typically downloads generic binaries
- Debian's APT provides [apt-build](https://manpages.debian.org/latest/apt-build/apt-build.1.en.html) to build packages with CPU architecture optimizations
- [Clear Linux](https://www.clearlinux.org/)'s swupd provides packages optimized for Intel CPU architectures
- [Spack](https://spack.readthedocs.io/en/latest/index.html) can be used to build and [containerize](https://spack.readthedocs.io/en/latest/containers.html) optimized software environments

---

## Multi-stage builds

- Singularity also supports [multi-stage builds](https://sylabs.io/guides/latest/user-guide/definition_files.html#multi-stage-builds)
- Primary use case is to minimize final image size by removing build dependencies
- In the first stage, build main application
    - Install build dependencies
    - Build main application
- In the second stage, copy build of main application
    - Starts with fresh software environment
    - No build dependencies needed
    - Only install run dependencies if needed

---

## Inspecting container images

See the help message for the image:

```bash
singularity inspect --helpfile python.sif
```

See the definition file used to build the image:

```bash
singularity inspect --deffile python.sif
```

See the environment variables set when running the container:

```bash
singularity inspect --environment python.sif
```

See the runscript for the image:

```bash
singularity inspect --runscript python.sif
```

---

## Singularity environment variables

- Many useful environment variables can be set
- Add to `~/.bashrc` to automatically set every time you log in
- See full list [here](https://docs.sylabs.io/guides/latest/user-guide/appendix.html#singularityce-s-environment-variables)

```bash
export SINGULARITY_TMPDIR=/dev/shm/$USER

export SINGULARITY_CACHEDIR=/scratch1/$USER/singularity

export SINGULARITY_CLEANENV=true

export SINGULARITY_NO_HOME=true
```

---

## Singularity documentation

- [Official docs](https://sylabs.io/guides/latest/user-guide/)
- Command-line help

```bash
singularity help

singularity help shell

singularity help exec

singularity help run
```

---

## Additional resources

- [Singularity website](https://sylabs.io/singularity)
- [Singularity documentation](https://sylabs.io/guides/master/user-guide/)
- [Singularity tutorial](https://sylabs.github.io/singularity101/)
- [Singularity Remote Builder](https://cloud.sylabs.io/builder)

Container registries:

- [Singularity Cloud Library](https://cloud.sylabs.io/library)
- [Docker Hub](https://hub.docker.com/)
- [BioContainers](https://biocontainers.pro)
- [Dockstore](https://www.dockstore.org/)
- [NVIDIA GPU Cloud Catalog](https://catalog.ngc.nvidia.com/containers)

---

## CARC support

- [CARC Singularity template definition files](https://github.com/uschpc/singularities)
- [Submit a support ticket](https://www.carc.usc.edu/user-support/submit-ticket)
- Office Hours
  - Every Tuesday 2:30-5pm
  - Get Zoom link [here](https://www.carc.usc.edu/services/education/office-hours-and-consultations.html)

---

## Exercises

- Pull a container image from a registry
- Build a custom container image
    - Create a definition file
    - Use the Remote Builder to build the custom image
    - Transfer the image to a CARC file system
- Inspect a container image
- Bind mount a `/project` directory to a container and list its files
- Run a script or program through a container