Container (Apptainer)#

Containerization allows you to stick your application and all of its dependencies into a single package. This makes the application portable, shareable and reproducible across different computing platforms and environments. It also allows for a faster loading of your setup, when used as a replacement for conda or python environments.

At NHR@FAU, Apptainer (formerly known as Singularity) is the standard container solution. It is specifically designed for HPC systems and causes no performance penalties.

Info

Docker containers cannot be run natively on the HPC system. However, they can be easily converted to the Apptainer format.

Properties of Apptainer:

available on all NHR@FAU systems
all filesystems available by default inside the container
supports GPU-dependent applications
not suitable for MPI applications across multiple nodes

Basic usage#

Command	Explanation
`apptainer build <container_name> <def-file>`	Build an Apptainer image
`apptainer pull <URI>`	Pull an image from a URI
`apptainer push <URI>`	Upload image to the provided URI
`apptainer run <container_name>`	Run the user-defined default command within a container
`apptainer exec <container_name> <command>`	Run a command within a container
`apptainer inspect <container_name>`	Check container metadata

More information including examples of all basic commands and workflows is available in the Apptainer Quick Start Guide.

Using existing containers#

Download / pull a container from a container repository (DockerHub) and it will be automatically converted into the Apptainer (.sif) format: apptainer pull docker://<repository>
Enter container with a shell: apptainer shell <container_name>
Execute commands inside a container: apptainer exec <container_name> <command>
Run pre-defined runscript of container: apptainer run <container_name> or ./<container name>

Building your own container#

Containers have to be built on a system where you have enough permissions and the security model allows it. Containers can be build on the AlmaLinux based cluster frontend nodes (Fritz, Alex, Helma, Woody, Meggie). It is not possible to build them on our Ubuntu based systems (TinyX, Testcluster), but we offer a work around by building the container inside a container. If you get the following error

ERROR  : Could not write info to setgroups: Permission denied
ERROR  : Error while waiting event for user namespace mappings: Success
FATAL:   While performing build: while running engine: exit status 1

try replacing apptainer build <options> with /apps/singularity/apptainer-wrapper.sh build <options>

Info

They can also be built on your local machine making permissions and security model less of an issue.

Interactive build - example#

Create sandbox based on a pre-existing container: apptainer build --sandbox <sandbox_name> docker://<repository>
Enter (writable) container with shell: apptainer shell --writable <sandbox_name>
Install/setup software inside container
Convert sandbox to image and back again: apptainer build <container_name>.sif <sandbox_name>, apptainer build --sandbox <sandbox_name> <container_name>.sif

Info

On Helma, sandbox containers should be prepared in $TMPDIR. In workspaces, the inodes are not sufficient and in $HOME it crowds the space.

Build from definition file#

Definition specify how a container is build and configured and therefore help to make the build reproducible. Apptainer definition files and Dockerfiles are very similar in how they work, but they use different syntax.

To build a container from a definition file, use the following command:

apptainer build <container_name>.sif <definition_file>

A simple example for a definition file is given below:

Python example#

Bootstrap: docker
From: ubuntu:latest

%post
  apt-get dist-upgrade
  apt-get update
  apt-get install -y python
  mkdir /test
  mv /python_sum.py /test

%files
python_sum.py

%runscript
exec "python" "/test/python_sum.py" "$@"

A definition file contains a bootstrap header where you choose the base container and sections where you install your software. In most cases, you want to start from an existing image from DockerHub. In this example, an Ubuntu image is chosen.

In the %post section, you can specify what happens during build time in the container, e.g. downloading and installing software and libraries, creating directories, etc.

The %files section can be used to copy files into the container. This is executed before the %post section.

The %runscript is the command that is executed when the container image is run via apptainer run.

The %environment section can be used to set environment variables inside the container. This is executed before the %runscript section.

More information on definition files and the different sections is available in the Apptainer documentation.

python container with conda example#

In this example, a definition file is shown to setup a python environment from a requirements.txt file using conda. If you need packages, that are not installed in the original docker image, you can install them with apt-get during the build.

Bootstrap: docker
From: dockerhub-mirror3.rrze.uni-erlangen.de/condaforge/miniforge3:latest

%files
    requirements.txt /

%environment
    export PATH=/opt/conda/envs/myenv/bin:$PATH
    export CONDA_DEFAULT_ENV=myenv

%post
    apt-get update -y
    apt-get install -y <packages>
    apt-get clean

    # Create a new conda environment named 'myenv' with Python installed
    conda create -y -n myenv python=3.11

    # Activate the conda environment
    . /opt/conda/etc/profile.d/conda.sh
    conda activate myenv

    # Install pip dependencies inside the conda environment
    pip install --upgrade pip
    pip install -r /requirements.txt

    # Optional: clean up
    conda clean --all -y

%runscript
    exec "$@"

Depending on your task, you may change the following settings:

Base container: Best choose the container that is fundamental to your environment. Use condaforge/miniforge3 if you already have a conda environment definition file and install it directly with conda env create -f /environment.yaml. Use the nvidia/cuda container if you need some special cuda libraries. When needing pure pytorch, you may use the 'pytorch/pytorch' docker container and use pip to install all your python packages.
Packages: If you want to use specific packages that are not present in the base container. You cannot use module load inside of the container and you are responsible for installing all required programs with apt-get.

Using GPUs inside containers#

Apptainer natively supports running GPU-enabled applications inside a container.

On the GPU clusters (Alex, Helma, TinyGPU), GPU device libraries are automatically bind-mounted into the container. No additional options should be necessary.

Requirements:

Host has working installation of GPU driver and CUDA libraries (Alex, Helma, TinyGPU)
CUDA version of application inside container must be compatible with host installation

If you encounter problems with missing GPU-support, try the Apptainer commands run/shell/execute with the --nv option, e.g. apptainer run --nv <container_name>.

Additional hints#

Per default, all file systems (/home, ...) are mounted inside a container.
- To prevent mounting any file systems: apptainer run --contain <container_name>
- Specify different home directory: apptainer run -H $HOME/my-container-home <container_name>
Pulled container images (e.g. from DockerHub) are by default saved to $HOME/.apptainer/cache. Set the environment variable $APPTAINER_CACHEDIRto different location, e.g. $WORK to save space in $HOME.
Using MPI inside containers is not recommended, as it requires the exact version of the host MPI implementation including all dependencies (e.g. Slurm, Infiniband libraries, ...) to work properly.
To efficiently use RDMA/Infiniband communication, make sure to include rdma-cora, libibverbs1, etc. in your image. Check (debug) the output (see Debugging NCCL). For NCCL as used by Pytorch, etc. check for an error such as NCCL INFO Failed to open libibverbs.so[.1].