Singularity/Apptainer update

During the October 26, 2022 maintenance window, Singularity was removed and replaced with Apptainer. The commands singularity (now a link pointing to Apptainer) and apptainer may be used to perform all the same operations you're used to and you can still run your existing images. However, remote builds via SyLabs are no longer supported. Instead, in many cases you may build your image directly on a compute node using:

$ singularity build local_image.sif container.recipe

Containers Overview

A container is a packaged unit of software that contains code and all its dependencies including, but not limited to: system tools, libraries, settings, and data. This makes applications and pipelines portable and reproducible, allowing for a consistent environment that can run on multiple platforms.

Shipping containers have frequently been used as an analogy because the container is standard, does not care what is put inside, and will be carried on any ship; or in the case of computing containers, it can run on many different systems.

Docker is widely used by researchers, however, Docker images require root privileges which means they cannot be run in an HPC environment.

Apptainer (formerly Singularity) addresses this by completely containing the authority so that all privileges needed at runtime stay inside the container. This makes it ideal for the shared environment of a supercomputer. Even better, a Docker image can be encapsulated inside an Apptainer image. Some ideal use cases that can be supported by Apptainer on HPC include:

You already use Docker and want to run your jobs on HPC.
You want to preserve your environment so a system change will not affect your work.
You need newer or different libraries than are offered on the system.
Someone else developed a workflow using a different version of Linux.
You prefer to use a Linux distribution other than CentOS (e.g. Ubuntu).
You want a container with a database server like MariaDB.

The documentation here provides instructions on how to either take a Docker image and run it from Apptainer, or create an image using Apptainer only.

Contents

Accessing Apptainer on HPC

Apptainer is installed on the operating systems of all HPC compute nodes, so can be easily accessed either from an interactive session or batch script without worrying about software modules.

Building a Container

With the introduction of Apptainer during the October 26, 2022 maintenance cycle, remote builds on SyLabs are no longer supported. Instead, in most cases it should be possible to build your images directly on a compute node using:

$ apptainer build local_image.sif container.recipe

This has been tested for recipes bootstrapping off of Docker images. We have found that in some cases (e.g. Boostrap: yum images) a local build will fail due to permissions issues. If you experience this and need assistance, contact our consultants and they can help come up with some alternatives.

Apptainer, Nvidia, and GPUs

One of the most significant use cases for Apptainer is to support machine learning workflows. For information on using GPUs on HPC, see our GPU documentation.

Pulling Nvidia Images

The NVIDIA GPU Cloud (NGC) provides GPU-accelerated HPC and deep learning containers for scientific computing. NVIDIA tests HPC container compatibility with the Singularity runtime through a rigorous QA process. Application-specific information may vary so it is recommended that you follow the container-specific documentation before running with Singularity. If the container documentation does not include Singularity information, then the container has not yet been tested under Singularity. Apptainer can be used to pull, execute, and bootstrap off of Singularity images.

Pulling Images Instructions

The containers from Nvidia that are in /contrib have been modified to include path bindings to /xdisk and /groups. They also include the path to the Nvidia commands like nvidia-smi.
Because login nodes are small and do not provide software, singularity images should be pulled and executed on a compute node.

To start, you'll need to register with Nvidia. Once you have an account, you can view their images from their catalogue. Click on the name of the software you're interested in to view available versions

If you click on the Tags tab at the top of the screen, you'll find the different versions that are available for download. For example, if we click on TensorFlow, we can get the pull statement for the latest tag of TensorFlow 2 by clicking the ellipses and selecting Pull Tag.

This will copy a docker pull statement to your clipboard, in this case:

$ docker pull nvcr.io/nvidia/tensorflow:22.02-tf2-py3

To pull and convert this NGC image to a local Apptainer image file, we'll convert this to:

$ apptainer build ~/tensorflow2-22.02-py3.sif docker://nvcr.io/nvidia/tensorflow:22.02-tf2-py3

The general format for any pull you want to do is:

$ apptainer build <local_image_name> docker://nvcr.io/<registry>/<app:tag>

This Apptainer build command will download the app:tag NGC Docker image, convert it to Apptainer format, and save it to the local filename local_image_name.

Running Nvidia Images

Directory access:

Apptainer containers are themselves ostensibly read only. In order to provide application input and output host directories are generally bound to the container, this is accomplished through the Apptainer -B flag. The format of this flag is -B <host_src_dir>:<container_dst_dir>. Once a host directory, host_src_dir, is bound into the container you may interact with this directory from within the container, located at container_dst_dir, the same as you would outside the container.

GPU support:

All NGC containers are optimized for NVIDIA GPU acceleration so you will always want to add the --nv flag to enable NVIDIA GPU support within the container.

Standard run command:

The Apptainer command below represents the canonical form that will be used on the Ocelote cluster.

$ singularity exec --nv --pwd <work_dir> <image.simg> <cmd>   # <work_dir> should be set to either $HOME or /tmp