Binary deployments with Spack

Table of content

Spack (https://spack.io) is a package manager for Linux that makes it easy to deploy scientific software on computers. One of the main differences from apt or dnf is that Spack can be installed at the user level, rather than globally by the system administrator. To install Spack, you only need to clone a GitHub repository, which can then be loaded and used to install packages that are usually installed at the system level, such as GCC, CUDA, and others.

Spack is also a source-based package manager. This means that the primary method of installing packages is to have them built from source by Spack itself. For example, if we want to install CMake, Spack will fetch the source code and build it locally — along with every dependency of CMake. In contrast, a binary-based package manager like APT will download the pre-built CMake from Debian’s servers, which was compiled in their server farm. Source-based package managers have several advantages over binary-based ones, especially in the context of supercomputers:

Optimization flags: When building C/C++ packages, you can apply compiler flags to optimize the build for specific CPU microarchitectures. This may make a binary for x86-64 unusable on other x86-64 machines if the compiler adds CPU instructions that are not present on older processors. Binary-based package managers must make a compromise by ensuring their pre-built packages are generic enough for many CPU variants, while sacrificing potential performance.
Fine control of version dependencies: Scientific software often relies on specific versions of libraries. A source-based package manager allows you to rebuild all reverse-dependencies of a package when you change its version, while you are very limited when using a binary-based one.
Different variants of the same package: Similar to having multiple versions of the same package, packages can be compiled with different sets of features, such as support for CUDA or ROCm, support for specific XML libraries, etc. This can be solved by binary-based package managers by providing multiple package variants that conflict with each other, but it is more cleanly solved by configuring your package variants in Spack.

On the other hand, one of the main issues with source-based package managers, including Spack, is the time required to compile packages. Because the entire dependency chain must be built from source, it can take a considerable amount of time and compute resources to prepare your toolchain. This can be an even worse problem when iterating on different package variants or versions to configure your environment.

To overcome this problem, this post discusses how to bridge the best of both worlds for Spack package management: using a binary cache. By using a binary cache, we can “pre-build” packages for an environment on our cloud platform, so that when we call spack install, it will attempt to use pre-built packages. Spack remains a source-based package manager, and if it doesn’t find a binary for a specific package, it can still build it from source.

By using a binary cache, we can provide users with an easy onboarding experience with Spack while retaining the features that make Spack useful, such as changing package variants, versions, or optimization levels.

Using a binary cache

Before discussing how to set up a binary cache, let’s see how the binary cache is used from a user’s perspective.

Tip

In Spack, we will see the name “mirror” to refer to binary caches.

$ spack mirror --help
usage: spack mirror [-hn] SUBCOMMAND ...

manage mirrors (source and binary)

positional arguments:
  SUBCOMMAND
    create           create a directory to be used as a spack mirror, and fill it with package archives
    destroy          given a url, recursively delete everything under it
    add              add a mirror to Spack
    remove (rm)      remove a mirror by name
    set-url          change the URL of a mirror
    set              configure the connection details of a mirror
    list             print out available mirrors to the console

options:
  -h, --help         show this help message and exit
  -n, --no-checksum  do not use checksums to verify downloaded files (unsafe)

To add a binary cache, the user simply needs to run a command. It is also possible to include the mirror as part of a Spack environment instead:

$ spack mirror add --unsigned inria-mirror oci://registry.gitlab.inria.fr/numpex-pc5/wp3/spack-stack/buildcache-rhel-8

# or in your environment's spack.yaml
spack:
  mirrors:
    inria-mirror:
      url: oci://registry.gitlab.inria.fr/numpex-pc5/wp3/spack-stack/buildcache-rhel-8
      signed: false

Finally, to install packages from the mirror, the user simply calls spack install as usual. As Spack installs packages, it will check if the package is cached in the mirror. If it is, it will download it instead of building from source.

It is important to understand that Spack checks if a package is cached by comparing the package’s spec hash. For example:

$ spack spec -L cmake | grep cmake
 -   gq5bqyhsvdlqs6qqjvs6useiq6kzhakf  cmake@3.31.6~doc+ncurses+ownlibs~qtgui build_system=generic build_type=Release arch=linux-ubuntu24.04-icelake %c,cxx=gcc@13.3.0

Spack will only download a CMake package from the cache if there is one with the same hash. The hash is calculated from the version of CMake, the build flags, the system architecture, and the hashes of all of its dependencies.

Important

This means packages in a binary cache must match the user’s system to be downloadable (Debian 11, RHEL 9, etc).

Creating a binary cache

To populate a binary cache, it is important to identify:

Where to store the packages
Where to build the packages

Package storage and serving

To serve packages from a binary cache, Spack doesn’t use any custom-made system that requires you to host a solution. Instead, it relies on a Docker Container Registry. To be clear: we don’t do anything related to containers. Spack pushes packages as if they were container layers to a container registry and pulls them transparently.

The benefit of this approach is that there are many cloud providers that offer container registry solutions:

GitHub with the GitHub Container Registry: https://docs.github.com/en/packages/working-with-a-github-packages-registry/working-with-the-container-registry
GitLab and its container registry: https://docs.gitlab.com/user/packages/container_registry
AWS https://aws.amazon.com/en/ecr and other cloud providers

The URL format for the Spack cache would be:

For GitHub: oci://ghcr.io/<username>/<repository>/<mirror name>
For GitLab (self-hosted): oci://<hostname>/numpex-pc5/<group>/<subgroup>/<repository>/<mirror name>

To be able to push to the cache, you will need a username and password. Since these are the same credentials you would use for the container registry of your choice, there should be documentation available for it. You can configure the credentials for the mirror as follows:

$ spack mirror add \
  --unsigned \
  --oci-username-variable OCI_USERNAME_VARIABLE \
  --oci-password-variable OCI_PASSWORD_VARIABLE \
  name \
  url

We have a quick guide to configure the built-in container registry for GitLab in the following link: Setup a Container Registry on GitLab.

Package building

After we have configured our mirror to push packages to, we need to build the packages themselves. The first option is to build them locally on your PC for debugging purposes. This is useful for getting comfortable with the tools and checking that pushing and pulling work as expected. Packages can be pushed with the following command:

$ spack buildcache push <mirror name> <specs...>

To automate this process, we can use a CI solution like GitHub Actions or GitLab pipelines. The checklist of elements you will need to consider includes:

Which packages to build
Committing the lockfile
Which microarchitecture to target
Which operating system to target
Configuring the padded_length

A simple Spack environment like the following should be enough to get started:

# spack.yaml
spack:
  view: true
  concretizer:
    unify: true

  # Declare which packages to be cached
  specs:
  - cmake
  - python

  # Set the padding length to a high value.
  config:
    install_tree:
      padded_length: 128

  # Declare the target microarchitecture
  packages:
    all:
      require:
      - x86_64_v2

We mark all packages to target x86_64_v2 (or whichever architecture you want to target) so that Spack doesn’t try to autodetect the architecture of the host system, but rather uses the explicitly set value. Otherwise, you may encounter the following problem:

GitHub Actions concretizes the environment to x86_64_v4 because of the CI system
You download the environment to an older machine
Programs crash due to missing instructions

As for padded_length: when you build a package, the build system will insert references to the absolute paths of other packages it depends on. The problem is that Spack can be installed to any path, so it must perform relocation.

Relocation is a process where a pre-built package gets its references replaced, for example /home/runner/spack/bin/cmake → /home/myuser/spack/bin/cmake. When dealing with an actual binary, this string will be embedded at some location:

                   0 1 2 3 4 5 6 7                 0 1 2 3 4 5 6 7
   0 1 2 3 4 5 6 7                 0 1 2 3 4 5 6 7
0  x x x x x x x x / h o m e / r u n n e r / s p a c k / b i n / c
1  m a k e 0 x x x x x x x x x x x x x x x x x x x x x x x x x x x

The problem arises when you want to replace it with a string longer than the original: you cannot assume that there will be free space after the original string. Therefore, the only operation you can safely perform is to shrink a string. By using padded_length, Spack will artificially create paths with a specified number of padding characters, so that if the number is large enough, we can assume that paths will always be shortened. For example, it will install packages to /home/runner/spack/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/bin/cmake instead (I simplified the actual installation paths for clarity).

Regarding the spack.lock file, to enable users to easily download from the cache, I would recommend committing and sharing the spack.lock with users of the cache. The two possibilities are:

CI builds packages, but the spack.lock is not preserved: when users concretize the environment on their own, they may or may not get the same versions and hashes that are actually cached, since a spack.yaml doesn’t guarantee reproducibility.
CI builds packages and pushes the spack.lock back into the environment: users won’t have to re-concretize the environment, so they will get the hashes for the packages that were built and cached by CI.

To preserve the spack.lock from CI, your caching workflow might look like this:

Set up Spack
Concretize the environment, rewriting the spack.lock if needed
Build and push the environment
Commit and push the spack.lock back to the repository

Finally, when populating a Spack binary cache, it is important to consider the Linux distribution to target. Since Spack doesn’t bootstrap itself from glibc, it actually links its packages to the system’s glibc. This adds an implicit dependency on the system and is reflected in the package spec. For example, if I concretize a package:

$ spack spec -L cmake | grep cmake
 -   gq5bqyhsvdlqs6qqjvs6useiq6kzhakf  cmake@3.31.6~doc+ncurses+ownlibs~qtgui build_system=generic build_type=Release arch=linux-ubuntu24.04-icelake %c,cxx=gcc@13.3.0

This package is concretized for ubuntu24.04. Therefore, if you want to deploy pre-built Spack packages for an OS like Debian 11, you must ensure that you concretize and build the environment on Debian 11 as well.

This can be easily accomplished with a container, and both GitHub and GitLab provide means of running action steps under a container:

# github workflow
jobs:
  main:
    container: debian:11
  #...

# gitlab pipeline
main:
  image: debian:11
  # ...

Need more help?

If you need more help with creating your Spack binary cache to accelerate deployments on HPC centers, please get in contact with us !