Environment managment

Module System#

On HPC (High Performance Computing) clusters, software is often managed with modules.
Modules let you load and unload software without needing to install it yourself.

example

# Check available modules
module avail

# Load a specific software
module load python/3.10

# Unload it
module unload python/3.10

This ensures you use the correct version of software for your project.

Containers#

Containers provide reproducible environments: all dependencies, libraries, and software versions are packaged together. Docker & Apptainer/Singularity are the most known but it exists plenty of other container engine. If you are a bioconda user, all tools existing in bioconda have its counterpart as container in biocontainers.

Docker:

Convenient on personal computers (Linux, macOS, Windows with Docker Desktop).
Easy to use locally.

Apptainer (formerly Singularity):

Preferred on HPC systems because it does not require admin privileges.
Runs Docker images seamlessly on clusters.

Where to find containers?#

Containers can be found on registry/platforms, which hosts a wide variety of ready-to-use images for different software and environments. One of the best place is Quay.io that contains all container from biocontainers and much more!

How to get the container?#

To retrieve a container, you will have to use a pull command with the URL to the container you want to use.

Docker:

docker pull quay.io/biocontainers/samtools:0.1.19--h96c455f_14

Apptainer:

singularity pull docker://quay.io/biocontainers/samtools:0.1.19--h96c455f_14

How to run a containerized tool?#

Docker#

To start a container you will use the docker run command. A reallistic usage will imply different options e.g. to get an interactive shell inside the container:

docker run -it -v /host/path:/container/path container /bin/bash

option	explanation
--rm	tells Docker to automatically remove the container when it stops
-it	interactive + terminal
-v	/host/path:/container/path maps your host folder to container
/bin/bash	gives you an interactive shell inside the container. Otherwise Docker will run the default command in the image (defined by CMD in the Dockerfile) if any.

Tips

If you know the command you want to run within the container, no need of interactive shell (/bin/bash) you can instead directly call the exact command expected from within the container.
⚠️ Docker do not mount any local data within the container, you will have to use -v

monitoring docker container:

command	explanation
docker ps	List running containers
docker stop	Stop a container
docker rm	Remove a container

Singularity#

To start a shell inside the container with singularity run

singularity shell container.sif

Unlike Docker, singularity mount the working directory within the container. If you need to get access to data localised elsewhere, you will have to use the --bind /host/path:/container/path option.

Tips

If you know the command you want to run within the container, no need of interactive shell (singularity shell) you can instead directly call the exact command expected from within the container using singularity exec. e.g:
```
# Run a program inside an Apptainer container
singularity exec container.sif tool --help
```