Modern HPC Workflow Example (Guix)

Table of content

This guide applies the workflow presented at Modern HPC Workflow with Containers to run an application container built with Guix.

  • This tutorial will focus on using Grid5000 for both building the container with Guix and deploying it with Singularity, as it provides both tools.

  • The container may be built on any computer with Guix installed. You may refer to the documentation if you wish to install Guix on your machine. Beware that if you build it on your local machine, you’ll have to copy it to Grid5000.

  • Additional instructions will be provided for deployment on Jean-Zay, that can be easily adapted to any cluster supporting Singularity and using SLURM as job management system.

  • The application chosen as an example is Chameleon, a dense linear algebra software for heterogeneous architectures that supports MPI and NVIDIA GPUs through CUDA or AMD GPUs through ROCm.

Chameleon on NVIDIA GPUs

Build the container on Grid5000

  1. Login to Grid500 (detailed instructions here). The full list of resources shows where to find an NVIDIA GPU and an x86_64 CPU (for Singularity). For instance, the chifflot queue, located in Lille, contains nodes with NVIDIA P100 GPUs.
ssh lille.g5k
mkdir tuto && cd tuto
  1. Get the channels file. The chameleon-cuda package (the chameleon package variant with CUDA support) is defined in the Guix-HPC non-free channel, which is not activated by default.
wget https://numpex-pc5.gitlabpages.inria.fr/tutorials/hpc-env/workflow-example/channels.scm

The channels.scm file contains the following:

(list
  (channel
    (name 'guix-hpc)
    (url "https://gitlab.inria.fr/guix-hpc/guix-hpc.git")
    (branch "master")
    (commit
      "ae4a812197a7e565c22f763a1f09684257e79723"))
  (channel
    (name 'guix-hpc-non-free)
    (url "https://gitlab.inria.fr/guix-hpc/guix-hpc-non-free.git")
    (branch "master")
    (commit
      "c546e8121b4d8a715dcb6cd743680dd7eee30b0e"))
  (channel
    (name 'guix)
    (url "https://git.guix.gnu.org/guix.git")
    (branch "master")
    (commit
      "97fb1887ad10000c067168176c504274e29e4430")
    (introduction
      (make-channel-introduction
        "9edb3f66fd807b096b48283debdcddccfea34bad"
        (openpgp-fingerprint
          "BBB0 2DDF 2CEA F6A8 0D1D  E643 A2A0 6DF2 A33A 54FA"))))
  (channel
    (name 'guix-science-nonfree)
    (url "https://codeberg.org/guix-science/guix-science-nonfree.git")
    (branch "master")
    (commit
      "de7cda4027d619bddcecf6dd68be23b040547bc7")
    (introduction
      (make-channel-introduction
        "58661b110325fd5d9b40e6f0177cc486a615817e"
        (openpgp-fingerprint
          "CA4F 8CF4 37D7 478F DA05  5FD4 4213 7701 1A37 8446"))))
  (channel
    (name 'guix-science)
    (url "https://codeberg.org/guix-science/guix-science.git")
    (branch "master")
    (commit
      "4ace0bab259e91bf0ec9d37e56c2676b1517fccd")
    (introduction
      (make-channel-introduction
        "b1fe5aaff3ab48e798a4cce02f0212bc91f423dc"
        (openpgp-fingerprint
          "CA4F 8CF4 37D7 478F DA05  5FD4 4213 7701 1A37 8446"))))
  (channel
    (name 'guix-past)
    (url "https://codeberg.org/guix-science/guix-past.git")
    (branch "master")
    (commit
      "70fc56e752ef6d5ff6e1e1a0997fa72e04337b24")
    (introduction
      (make-channel-introduction
        "0c119db2ea86a389769f4d2b9c6f5c41c027e336"
        (openpgp-fingerprint
          "3CE4 6455 8A84 FDC6 9DB4  0CFB 090B 1199 3D9A EBB5")))))
  1. Generate the Singularity container image with the guix pack command, prefixed with guix time-machine in order to use our channels.scm file. The -r option creates a symbolic link to the resulting container image in the Guix store, as chameleon.sif.
guix time-machine -C channels.scm -- pack -f squashfs chameleon-cuda bash -r ./chameleon.sif
Tip

guix pack can generate different formats, like Singularity (squashfs), Docker or relocatable binaries.

Tip

Singularity needs bash to be in the package list.

Deploy the container on Grid5000

  1. Start an interactive job using OAR.
oarsub -I -p chifflot -l host=2
export LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libcuda.so
export OPENBLAS_NUM_THREADS=1
mpirun -machinefile $OAR_NODEFILE \
       --bind-to board \
       singularity exec\
                   --bind /usr/lib/x86_64-linux-gnu/:/usr/lib/x86_64-linux-gnu/ \
                   --bind /tmp:/tmp chameleon-cuda.sif \
                   chameleon_stesting -o gemm -n 4000 -b 160 --nowarmup -g 2
Tip

CUDA applications deployed with Guix need LD_PRELOAD to be set with the path to libcuda.so since the library is provided by the proprietary CUDA driver, installed on the machine, and not part of the Guix software stack.

Tip

The OPENBLAS_NUM_THREADS environment variable is set to improve the computation performance and not compulsory.

Deploy the container on Jean-Zay

  1. Copy the image to Jean-Zay. Depending on your SSH setup, you might have to adapt the commands below.
# Disconnect from Grid5000.
exit

# Copy the image from Grid5000 to Jean-Zay
scp lille.g5k:tuto/chameleon.sif jean-zay:chameleon.sif
  1. Setup the container image on Jean-Zay. First, the image has to be copied to the allowed space ($SINGULARITY_ALLOWED_DIR) in order to be accessible to Singularity. This step is specific to Jean-Zay, more details in the documentation. Then the singularity module needs to be loaded (this step is not always necessary, depending on the supercomputer, but is not specific to Jean-Zay).
ssh jean-zay
idrcontmgr cp chameleon.sif
module load singularity
  1. Start the job using the container with SLURM.
OPENBLAS_NUM_THREADS=1 \
  srun -A account@v100 \
       --time=0:10:00 \
       --nodes=2 \
       --cpu-bind=socket \
       --exclusive \
       --hint=nomultithread \
       --gres=gpu:4 \
       --mpi=pmi2 \
       singularity \
         exec --nv $SINGULARITY_ALLOWED_DIR/chameleon.sif \
         bash -c "LD_PRELOAD=/.singularity.d/libs/libcuda.so chameleon_stesting -o gemm -n 40000 -b 2000 --nowarmup -g 4"
Tip

Environment variables are propagated to the Singularity container context, but since the path to libcuda.so doesn’t exist outside of the container context (the path in bind-mounted by Singularity due to the --nv flag) it leads to an error when LD_PRELOAD is declared outside of the container context.

Deploy the container on Vega (EuroHPC)

  1. Copy the image to Vega. Depending on your SSH setup, you might have to adapt the commands below.
# Copy the image from Grid5000 to Vega
scp lille.g5k:tuto/chameleon.sif vega:chameleon.sif
  1. Start the job using the container with SLURM.
srun --exclusive \
     --partition=gpu \
     --gres=gpu:4 \
     -N 2 \
     --mpi=pmi2 singularity exec --bind /tmp:/tmp chameleon-cuda.sif \
     bash -c "LD_PRELOAD=/.singularity.d/libs/libcuda.so chameleon_stesting -o gemm -n 96000 -b 2000 --nowarmup -g 4"

Deploy the container on MeluXina (EuroHPC)

  1. Copy the image to MeluXina.
# Copy the image from Grid5000 to Jean-Zay
scp lille.g5k:tuto/chameleon.sif meluxina:chameleon.sif
  1. Start an interactive allocation with SLURM and load Singularity/Apptainer. On MeluXina, the singularity command is available through a module and the module command is only accessible on a compute node.
[login] srun --pty \
             -A project_id \
             --partition=gpu \
             -N 1 \
             --exclusive \
             --gpus-per-task=4 \
             --time=00:10:00 \
             --qos=test \
             bash
[compute] module load Apptainer/1.3.1-GCCcore-12.3.0
  1. Start the computation using Singularity.
[compute] OPENBLAS_NUM_THREADS=1 \
            exec \
            --nv \
            --bind /tmp:/tmp \
            chameleon-cuda.sif \
            bash -c 'LD_PRELOAD=/.singularity.d/libs/libcuda.so chameleon_stesting -o gemm -n 20000 -b 2000 --nowarmup -g 4'
Tip

In this example, we use a single node. In order to use multiple nodes, a script should be submitted using sbatch (not covered in this tutorial).

Build a Docker image on Grid5000

  1. Build a Docker container on Grid5000.
ssh lille.g5k
guix time-machine -C channels.scm -- pack -f docker chameleon-cuda bash -r ./chameleon.tar.gz

Deploy a Docker container on Irene (TGCC)

  1. Copy the container to Irene. Depending on your SSH setup, you might have to adapt the commands below.
# Disconnect from Grid5000.
exit

# Copy the image from Grid5000 to Jean-Zay
scp lille.g5k:chameleon.tar.gz irene:
  1. **SSH to Irene and import the image using pcocc-rs.
[irene-login] pcocc-rs image import docker-archive:chameleon.tar.gz chameleon
Tip

The TGCC uses a specific tool to deploy Docker images called pcocc-rs. See the documentation.

  1. Start a job in interactive mode.
[irene-login] ccc_mprun \
                -p v100 \
                -N 4 \
                -n 4 \
                -c 40 \
                -E '--mpi=pmi2' \
                -m work,scratch \
                -A user_account \
                -T 600 \
                -C chameleon \
                -E '--ctr-module nvidia' \
                -- bash -c "LD_PRELOAD=/pcocc/nvidia/usr/lib64/libcuda.so chameleon_stesting -o gemm -n 20000 -b 2000 --nowarmup -g 4"
Tip

On Irene, resources are allocated using ccc_mprun. See the documentation. For instance, the -s option spawns an interactive session directly on a compute node.

Tip

On Irene, the number of allocated GPUs is directly related to the number of allocated cores on the node. Here, 20 cores are allocated on a V100 which contains 40 cores in total, so 50% of the GPUs available on the node (4 x V100) are allocated. See the documentation.

Tip

The --module nvidia option make the CUDA libraries available inside the image in the /pcocc/nvidia/usr/lib64 folder.

Chameleon on AMD GPUs

Build the image on Grid5000

  1. Connect to Grid5000 and build the Singularity container.
ssh lille@g5k
cd tuto
guix time-machine -C channels.scm -- pack -f squashfs chameleon-hip bash -r ./chameleon-hip.sif

Deploy on Adastra

  1. Copy the Singularity image to Adastra. Depending on your SSH setup, you might have to adapt the commands below.
# Disconnect from Grid5000.
exit

# Copy the image from Grid5000 to Adastra
scp lille.g5k:tuto/chameleon-hip.sif adastra:chameleon-hip.sif
Warning

Before being able to use a custom Singularity image, it has be manually copied to an authorized path by the support, which should be contacted by email. See the documentation.

  1. Start a job in interactive mode.
ssh adastra
OPENBLAS_NUM_THREADS=1 \
  srun --cpu-bind=socket \
       -A user_account \
       --time=0:10:00 \
       --constraint=MI250 \
       --exclusive \
       --nodes=4 \
       --mpi=pmi2 \
       singularity exec \
                   --bind /proc \
                   --bind /sys \
                   /opt/software/containers/images/users/cad15174/chameleon-hip.sif \
                   chameleon_stesting -o gemm -n 96000 -b 2000 --nowarmup -g 8

Deploy on LUMI

  1. Copy the Singularity image to LUMI. Depending on your SSH setup, you might have to adapt the commands below.
# Copy the image from Grid5000 to LUMI
scp lille.g5k:tuto/chameleon-hip.sif lumi:chameleon-hip.sif
  1. Start a job in interactive mode.
ssh lumi
OPENBLAS_NUM_THREADS=1 \
  srun --cpu-bind=socket \
       -A project_id \
       --threads-per-core=1 \
       --cpus-per-task=56 \
       --ntasks-per-node=1 \
       -N 4 \
       --time=00:05:00 \
       --partition=dev-g \
       --mpi=pmi2 \
       singularity exec \
                   --bind /sys:/sys \
                   chameleon-hip.sif \
                   chameleon_stesting -o gemm -n 40000 -b 2000 --nowarmup -g 8

Bonus: relocatable binaries

For machines where Singularity is not available (or you have to ask support to deploy your custom image), an alternative can be the relocatable binary archive. The command below generates an archive containing chameleon-hip for AMD GPUs that can be run on e.g. Adastra:

guix time-machine -C channels.scm -- pack -R -S /bin=bin -C zstd chameleon-hip -r chameleon-hip.tar.zst

This archive can then be uploaded to a supercomputer (e.g. Adastra) and deployed:

# Copy the archive to Adastra
scp chameleon-hip.tar.zst adastra:
# SSH into Adastra
ssh adastra
# Extract the archive into its own folder
[adastra] mkdir chameleon-hip && zstd -d chameleon-hip.tar.zst \
                              && tar xf  chameleon-hip.tar -C chameleon-hip
# Start the job
[adastra] OPENBLAS_NUM_THREADS=1 \
            srun --cpu-bind=socket \
                 -A cad15174 \
                 --time=0:10:00 \
                 --constraint=MI250 \
                 --exclusive \
                 --nodes=4 \
                 --mpi=pmi2 \
                 $CCFRWORK/chameleon-hip-common/bin/chameleon_stesting \
                              -o gemm -n 96000 -b 2000 --nowarmup -g 8