Skip to content

Container (Apptainer)#

Containerization allows you to stick your application and all of its dependencies into a single package. This makes the application portable, shareable and reproducible across different computing platforms and environments. It also allows for a faster loading of your setup, when used as a replacement for conda or python environments.

At NHR@FAU, Apptainer (formerly known as Singularity) is the standard container solution. It is specifically designed for HPC systems and causes no performance penalties.

Info

Docker containers cannot be run natively on the HPC system. However, they can be easily converted to the Apptainer format.

Suggestions

It is suggested to wrap your python environment into a container, because the virtual environments usually contain large number of inodes, leading to massive problems on the file system. More details can be found in slides from HPC Cafe of June 2025.

Properties of Apptainer:

  • available on all NHR@FAU systems
  • all filesystems available by default inside the container
  • supports GPU-dependent applications
  • not suitable for MPI applications across multiple nodes

Basic usage#

Command Explanation
apptainer build <container_name> <def-file> Build an Apptainer image
apptainer pull <URI> Pull an image from a URI
apptainer push <URI> Upload image to the provided URI
apptainer run <container_name> Run the user-defined default command within a container
apptainer exec <container_name> <command> Run a command within a container
apptainer inspect <container_name> Check container metadata
apptainer cache clean Clean caches from old builds

More information including examples of all basic commands and workflows is available in the Apptainer Quick Start Guide.

Using existing containers#

  • Download / pull a container from a container repository (DockerHub) and it will be automatically converted into the Apptainer (.sif) format: apptainer pull docker://<repository>
  • Enter container with a shell: apptainer shell <container_name>
  • Execute commands inside a container: apptainer exec <container_name> <command>
  • Run pre-defined runscript of container: apptainer run <container_name> or ./<container name>

Building your own container#

Containers have to be built on a system where you have enough permissions and the security model allows it. In the HPC system, Containers can be build on the AlmaLinux based cluster frontend nodes (Fritz, Alex, Helma, Woody, Meggie). It is not possible to build them on our Ubuntu based systems (TinyX, Testcluster), but we offer a work around by building the container inside a container (link).

There are two ways to build a container, interactively or from a definition file:

Info

The containers .sif can also be built on local machine with enough permission and security.

Interactive build#

  • Create sandbox based on a pre-existing container: apptainer build --sandbox <sandbox_name> docker://<repository>
  • Enter (writable) container with shell: apptainer shell --writable <sandbox_name>
  • Install/setup software inside container
  • Convert sandbox to image and back again: apptainer build <container_name>.sif <sandbox_name>, apptainer build --sandbox <sandbox_name> <container_name>.sif

Info

On Helma, sandbox containers should be prepared in $TMPDIR. In workspaces, the inodes are not sufficient and in $HOME it crowds the space.

Build from definition file#

Definition specify how a container is build and configured and therefore help to make the build reproducible. Apptainer definition files and Dockerfiles are very similar in how they work, but they use different syntax.

To build a container from a definition file, use the following command:

apptainer build <container_name>.sif <definition_file>
More examples can be found here. A definition file contains a bootstrap header where you choose the base container and sections where you install your software/packages. In most cases, you want to start from an existing image from DockerHub.

  • In the %post section, you can specify what happens during build time in the container, e.g. downloading and installing software and libraries, creating directories, etc.
  • The %files section can be used to copy files into the container. This is executed before the %post section.
  • The %runscript is the command that is executed when the container image is run via apptainer run.
  • The %environment section can be used to set environment variables inside the container. This is executed before the %runscript section.

More information on definition files and the different sections is available in the Apptainer documentation.

Depending on your task, you may change the following settings:

  • Base container: Best choose the container that is fundamental to your environment. Use condaforge/miniforge3 if you already have a conda environment definition file and install it directly with conda env create -f /environment.yaml. Use the nvidia/cuda container if you need some special cuda libraries. When needing pure pytorch, you may use the pytorch/pytorch docker container and use pip to install all your python packages.

  • Packages: If you want to use specific packages that are not present in the base container. You cannot use module load inside of the container and you are responsible for installing all required programs with apt-get.

Using GPUs inside containers#

Apptainer natively supports running GPU-enabled applications inside a container.

On the GPU clusters (Alex, Helma, TinyGPU), GPU device libraries are automatically bind-mounted into the container. No additional options should be necessary.

Requirements:

  • Host has working installation of GPU driver and CUDA libraries (Alex, Helma, TinyGPU)
  • CUDA version of application inside container must be compatible with host installation

If you encounter problems with missing GPU-support, try the Apptainer commands run/shell/execute with the --nv option, e.g. apptainer run --nv <container_name>.

Additional hints#

  • Per default, all file systems (/home, ...) are mounted inside a container.

    • To prevent mounting any file systems: apptainer run --contain <container_name>
    • Specify different home directory: apptainer run -H $HOME/my-container-home <container_name>

  • Pulled container images (e.g. from DockerHub) are by default saved to $HOME/.apptainer/cache. Set the environment variable $APPTAINER_CACHEDIRto different location, e.g. $WORK to save space in $HOME.

  • Using MPI inside containers is not recommended, as it requires the exact version of the host MPI implementation including all dependencies (e.g. Slurm, Infiniband libraries, ...) to work properly.

  • To efficiently use RDMA/Infiniband communication, make sure to include rdma-cora, libibverbs1, etc. in your image. Check (debug) the output (see Debugging NCCL). For NCCL as used by Pytorch, etc. check for an error such as NCCL INFO Failed to open libibverbs.so[.1].

Examples#

Basic container

A simple example for a definition file is given below:

Bootstrap: docker
From: ubuntu:latest

%post
apt-get dist-upgrade
apt-get update
apt-get install -y python
mkdir /test
mv /python_sum.py /test

%files
python_sum.py

%runscript
exec "python" "/test/python_sum.py" "$@"
Python container with Conda

In this example, a definition file is shown to setup a python environment from a requirements.txt file using conda. If you need packages, that are not installed in the original docker image, you can install them with apt-get during the build.

Bootstrap: docker
From: dockerhub-mirror3.rrze.uni-erlangen.de/condaforge/miniforge3:latest

%files
    requirements.txt /

%environment
    export PATH=/opt/conda/envs/myenv/bin:$PATH
    export CONDA_DEFAULT_ENV=myenv

%post
    apt-get update -y
    apt-get install -y <packages>
    apt-get clean

    # Create a new conda environment named 'myenv' with Python installed
    conda create -y -n myenv python=3.11

    # Activate the conda environment
    . /opt/conda/etc/profile.d/conda.sh
    conda activate myenv

    # Install pip dependencies inside the conda environment
    pip install --upgrade pip
    pip install -r /requirements.txt

    # Optional: clean up
    conda clean --all -y

%runscript
    exec "$@"
Python container with pytorch

In this example, a python container is built with pytorch. 

Bootstrap: docker
From: pytorch/pytorch:2.7.1-cuda11.8-cudnn9-runtime

%files
    requirements.txt /

%post
    pip3 install -r requirements.txt

    python3 --version
    python3 -c "import torch; print(torch.__version__)"
    python3 -c "import cv2"

%runscript
    exec python3 "$@"
Example requirements.txt 
pytorch-lightning
pandas
pyrootutils
Pillow
scikit-learn
timm
tensorboard
tensorboardX
optuna
matplotlib
seaborn
opencv-python-headless

Known issues#

ERROR : Could not write info to setgroups: Permission denied

If you get the following error 

ERROR  : Could not write info to setgroups: Permission denied
ERROR  : Error while waiting event for user namespace mappings: Success
FATAL:   While performing build: while running engine: exit status 1
try replacing apptainer build <options> with /apps/singularity/apptainer-wrapper.sh build <options>

Incorrect package version during runtime

By default, apptainer binds the /home into the container during runtime and this can override the path to site-packages. try adding the --contain to remove the default binding, i.e. apptainer run --contain .... If the real /home is still visible in the container, try to bind /home to some empty path, i.e. apptainer run --bind /tmp:/home ...