Guix and Spack for Application Deployment Across Supercomputers

Table of content
  • Day: Thursday, 6th of February, 2025
  • Time:
    • 10h-12h: General presentation
    • 13h-16h: Practice time
  • Speakers:
    • Fernando Ayats Llamas
    • Romain Garbage
    • Bruno Raffin (for the intro)

Abstract

The standard way to install a library or application on a supercomputer is to rely on modules to load most dependencies, then compile and install the remaining pieces of software manually. In many cases, this is a time-consuming process that needs to be revisited for each supercomputer. As part of NUMPEX software integration efforts, we advocate relying on advanced package managers to compile and deploy codes with their dependencies. Guix and Spack, the two package managers favored at NUMPEX, are backed by strong communities and have thousands of packages regularly tested on CI farms. They come with feature-rich CLIs enabling package tuning and transformations to produce customized binaries, including modules and containers. They carefully control the dependency graph for portability and reproducibility. However, most HPC users have not yet made package managers a central tool for controlling and deploying their software stack, and supercomputer centers usually do not install them for easy direct user-level access. The goal of this tutorial is to demystify their usage for HPC and show that they can indeed be used as a universal deployment solution. This tutorial will be split into two parts. The morning (10:00-12:00) will be dedicated to presentations and discussions. We will present how users can leverage Guix and Spack to deploy libraries (potentially with all necessary extra tools to provide a full-blown development environment) on different supercomputers (Jean-Zay, Adastra, CCRT), either directly if available on the machine or through containers generated with Spack and Guix, and discuss issues related to performance, portability, and reproducibility. Anyone who has dealt with HPC application installation should find useful material in this tutorial. No specific background in Spack or Guix is necessary, and this is not about learning Spack or Guix or how to package a given library (other dedicated tutorials will be scheduled later).

The afternoon will be organized as a hands-on session where interested users will experiment with Spack/Guix-based deployment. We will provide a base application and open accounts on Grid'5000 for all attendees (where Spack and Guix:s are installed), but users can also bring their own applications and try deployment on their preferred supercomputers. We will be available (via visio and chat) to assist you and answer questions throughout the afternoon.

Note that this tutorial is part of the NUMPEX software integration strategy backed by the Exa-DI WP3 team. Our ambition is to have all NUMPEX-related libraries packaged with Guix and Spack, make Guix/Spack-based deployment part of every developer’s arsenal, and work with computing centers to make Guix/Spack-based user-level software deployment as frictionless as possible.

Resources

Recordings:

Slides:

Written guides:

Contact: