Spack package manager#

Some users require special software for their HPC runs which is not provided by the computing center. In order to provide this special software, the HPC systems at NHR@FAU provide the Spack package manager in the module environment under the name user-spack.

The benefits when using Spack:

Easy to maintain separate versions of the same package
Separate installations when using different versioned or built dependencies

The benefits also have their downside. You partly have to very specific what to install with which dependencies.

The main Spack documentation (for the latest version) can be found here.

Basics#

Load the user-spack module:

$ module avail user-spack
----------------------------------- /apps/modules/data/via-spack -----------------------------------
user-spack/0.17.0  user-spack/0.17.1  user-spack/0.18.0  user-spack/0.19.1
$ module load user-spack

All software will be installed below $WORK/USER-SPACK

List available compilers

The NHR@FAU team provides some predefined compilers which are mostly available as distinct modules or are part of the base system installation. They are only loosely related to the packages used inside Spack, so the list of compilers does not change when you load a module before.

$ spack compilers
==> Available compilers
-- dpcpp almalinux8-x86_64 --------------------------------------
dpcpp@2023.0.0

-- gcc almalinux8-x86_64 ----------------------------------------
gcc@12.1.0  gcc@11.2.0  gcc@8.5.0

-- intel almalinux8-x86_64 --------------------------------------
intel@2021.8.0  intel@2021.7.0

-- oneapi almalinux8-x86_64 -------------------------------------
oneapi@2023.0.0

List available packages

$ spack list netcdf
==> 7 packages.
netcdf-c netcdf-cxx netcdf-cxx4 netcdf-fortran parallel-netcdf py-h5netcdf py-netcdf4

List installed packages

$ spack find openmpi
==> 4 installed packages
-- linux-almalinux8-zen / gcc@8.4.1 -----------------------------
openmpi@3.1.6

-- linux-almalinux8-zen / intel@2021.4.0 ------------------------
openmpi@3.1.6

-- linux-almalinux8-zen / oneapi@2021.4.0 -----------------------
openmpi@3.1.6

-- linux-centos7-haswell / gcc@4.8.5 ----------------------------
openmpi@4.1.2

Get information about a package

$ spack info numactl
AutotoolsPackage: numactl

Description:
NUMA support for Linux

Homepage: https://github.com/numactl/numactl

Preferred version: 
2.0.14 https://github.com/numactl/numactl/archive/v2.0.14.tar.gz

Safe versions: 
2.0.14 https://github.com/numactl/numactl/archive/v2.0.14.tar.gz
2.0.12 https://github.com/numactl/numactl/archive/v2.0.12.tar.gz
2.0.11 https://github.com/numactl/numactl/archive/v2.0.11.tar.gz

Deprecated versions: 
None

Variants:
None

Build Dependencies:
autoconf automake gnuconfig libtool m4

Link Dependencies:
None

Run Dependencies:
None

Get available versions for a package

$ spack versions kokkos
==> Safe versions (already checksummed):
develop master 3.6.00 3.5.00 3.4.01 3.4.00 3.3.01 3.3.00 3.2.01 3.2.00 3.1.01 3.1.00 3.0.00
==> Remote versions (not yet checksummed):
3.6.01

Install a package#

The most complicated part is the specification of the package for installation. Spack uses a special syntax.

See Simple package installation, Custom versions & configurations and Customize dependencies in the Spack docs for more information.

Check what will be installed#

You can user spack spec <pkgspec> to get the tree of required dependencies.

Example of spack spec

$ spack spec libvdwxc
Input spec
--------------------------------
libvdwxc

Concretized
--------------------------------
libvdwxc@0.4.0%gcc@8.5.0+mpi~pfft arch=linux-almalinux8-x86_64
    ^fftw@3.3.10%gcc@8.5.0+mpi~openmp~pfft_patches precision=double,float arch=linux-almalinux8-x86_64
    ^openmpi@3.1.6%gcc@8.5.0~atomics~cuda~cxx~cxx_exceptions~gpfs~internal-hwloc~java+legacylaunchers~lustre~memchecker+pmi+romio+rsh~singularity+static+vt+wrapper-rpath fabrics=psm2 schedulers=slurm arch=linux-almalinux8-x86_64
    ^hwloc@1.11.13%gcc@8.5.0~cairo~cuda~gl~libudev+libxml2~netloc~nvml~opencl+pci~rocm+shared patches=d1d94a4 arch=linux-almalinux8-x86_64
    ^libpciaccess@0.16%gcc@8.5.0 arch=linux-almalinux8-x86_64
        ^libtool@2.4.7%gcc@8.5.0 arch=linux-almalinux8-x86_64
        ^m4@1.4.19%gcc@8.5.0+sigsegv patches=9dc5fbd,bfdffa7 arch=linux-almalinux8-x86_64
        ^libsigsegv@2.13%gcc@8.5.0 arch=linux-almalinux8-x86_64
        ^pkgconf@1.8.0%gcc@8.5.0 arch=linux-almalinux8-x86_64
        ^util-macros@1.19.3%gcc@8.5.0 arch=linux-almalinux8-x86_64
    ^libxml2@2.9.13%gcc@8.5.0~python arch=linux-almalinux8-x86_64
        ^libiconv@1.16%gcc@8.5.0 libs=shared,static arch=linux-almalinux8-x86_64
        ^xz@5.2.5%gcc@8.5.0~pic libs=shared,static arch=linux-almalinux8-x86_64
        ^zlib@1.2.12%gcc@8.5.0+optimize+pic+shared patches=0d38234 arch=linux-almalinux8-x86_64
    ^ncurses@6.2%gcc@8.5.0~symlinks+termlib abi=none arch=linux-almalinux8-x86_64
    ^numactl@2.0.14%gcc@8.5.0 patches=4e1d78c,62fc8a8,ff37630 arch=linux-almalinux8-x86_64
    ^autoconf@2.69%gcc@8.5.0 patches=35c4492,7793209,a49dd5b arch=linux-almalinux8-x86_64
        ^perl@5.34.1%gcc@8.5.0+cpanm+shared+threads arch=linux-almalinux8-x86_64
        ^berkeley-db@18.1.40%gcc@8.5.0+cxx~docs+stl patches=b231fcc arch=linux-almalinux8-x86_64
        ^bzip2@1.0.8%gcc@8.5.0~debug~pic+shared arch=linux-almalinux8-x86_64
        ^diffutils@3.8%gcc@8.5.0 arch=linux-almalinux8-x86_64
        ^gdbm@1.19%gcc@8.5.0 arch=linux-almalinux8-x86_64
        ^readline@8.1%gcc@8.5.0 arch=linux-almalinux8-x86_64
    ^automake@1.16.5%gcc@8.5.0 arch=linux-almalinux8-x86_64
    ^opa-psm2@11.2.206%gcc@8.5.0+avx2 arch=linux-almalinux8-x86_64
    ^openssh@8.0p1%gcc@8.5.0 arch=linux-almalinux8-x86_64
    ^slurm@21.08.3-1%gcc@8.5.0~gtk~hdf5~hwloc~mariadb~pmix+readline~restd sysconfdir=PREFIX/etc arch=linux-almalinux8-x86_64

There you see the whole package tree including the used compilers, the feature flags and the architecture used. It does not show which packages are already installed and which need to be built.

Example from the Spack docs#

# Install hdf5 and link it with specific versions of openmpi and hwloc
$ spack install hdf5@1.10.1 %gcc@4.7.3 +debug ^openmpi+cuda fabrics=auto ^hwloc+gl

This installs the hdf5 package in version 1.10.1 with the gcc compiler in version 4.7.3 and enables the debug flag. As dependencies, it uses openmpi with cuda feature enabled and the fabrics variable set to auto and hwloc with the gl feature enabled.

Downloading sources & packages#

On some compute systems, it might be required to use a proxy server for accessing the internet.

$ export http_proxy=http://proxy:80
$ export https_proxy=http://proxy:80

It takes some time#

Spack itself is not the fastest piece of software and all operations at the computing centers are on remote file systems. Those file systems commonly don't like the traffic which happens when building software.

Moreover, Spack installs all dependencies down to the lowest level for each compiler&deps combination. So you might see builds of base packages like zlib, perl, ncurses, etc. The NHR@FAU team tries to provide a basic set of common packages to avoid to many builds of the same dependency over and over again.

Due to the high flexibility of packages and versions coupled together by Spack and the various compilers, it might be that a build fails. It often helps to add a dependency with ^ compiled with gcc. For the netcdf-fortran package in the next paragraph, the ^numactl@2.0.14%gcc dependency was required otherwise some dependency of OpenMPI failed with missing <numa.h>.

Using an installed package#

All installed packages (and installed dependencies) should be visible as modules after installation (see module avail). If not, try unloading the user-spack module and load it again. If it is still not visible, try logging in again in a new session.

$ module load user-spack
$ spack install netcdf-fortran%oneapi@2022.1.0 \
                ^netcdf-c%oneapi@2022.1.0 \
                ^openmpi@4.1.3%oneapi@2022.1.0 \
                ^cmake@3.23.1 \
                ^numactl@2.0.14%gcc
$ module avail netcdf-fortran
------------------- /home/saturn/unrz/unrzXXX/USER-SPACK/share/spack/modules/linux-almalinux8-x86_64 -------------------
netcdf-fortran/4.5.4-oneapi2022.1.0-openmpi-3bmi7ym

You can use the software versions installed through Spack like normal modules. The module files are managed by Spack but it has its own mechanism to setup the proper environment, it does not load the modules corresponding to the environment.

$ module load netcdf-fortran/4.5.4-oneapi2022.1.0-openmpi-3bmi7ym
$ nf-config

The module avail command shows all installed packages, also the base packages like zlib, perl, ncurses, etc. You can load them separately, but you do not need to load them explicitly to use netcdf-fortran (in this case).

module avail output after installing a package through Spack

$ module avail
------------------- /home/saturn/unrz/unrzXXX/USER-SPACK/share/spack/modules/linux-almalinux8-x86_64 -------------------
autoconf/2.69-gcc8.4.1-g4skt4y libbsd/0.11.3-gcc8.4.1-flcvhjw numactl/2.0.14-gcc12.1.0-elphffx
autoconf/2.69-gcc12.1.0-zp6lni4 libbsd/0.11.5-oneapi2022.1.0-csnexv2 numactl/2.0.14-oneapi2022.1.0-2rlabzl
autoconf/2.69-oneapi2022.1.0-ogngd4o libevent/2.1.12-oneapi2022.1.0-urjh4xz opa-psm2/11.2.206-gcc12.1.0-7aw5fpm
automake/1.16.3-gcc8.4.1-2mgywnr libiconv/1.16-gcc8.4.1-wk2vhbw opari2/2.0.6-gcc8.4.1-4vdn6tg
automake/1.16.5-oneapi2022.1.0-v63ckpi libiconv/1.16-oneapi2022.1.0-caghp63 opari2/2.0.6-intel2021.4.0-34c744c
berkeley-db/18.1.40-gcc8.4.1-egliouh libidn2/2.3.0-gcc8.4.1-rggatlp openmpi/4.1.3-oneapi2022.1.0-44wu2xf
berkeley-db/18.1.40-oneapi2022.1.0-emyiyb2 libidn2/2.3.0-oneapi2022.1.0-tjr7ymg otf2/2.3-gcc8.4.1-rjoiusx
bzip2/1.0.8-gcc8.4.1-ahvvlrw libmd/1.0.3-gcc8.4.1-kqgrzgj otf2/2.3-intel2021.4.0-inao6hn
bzip2/1.0.8-oneapi2022.1.0-md3kfxz libmd/1.0.4-oneapi2022.1.0-zeltuf3 papi/6.0.0.1-gcc8.4.1-pqalrnb
cmake/3.23.1-oneapi2022.1.0-afaxwof libpciaccess/0.16-oneapi2022.1.0-iyi6tbt papi/6.0.0.1-intel2021.4.0-zvpbzqb
[...]

To remove them from the environment, just unload the module

$ module unload netcdf-fortran

What about `spack load`, `spack unload` and `spack env`#

These sub-commands of Spack are part of a builtin module system. It is currently not usable at the HPC systems of NHR@FAU and HPC4FAU.

Spack package manager#

Basics#

Install a package#

Check what will be installed#

Example from the Spack docs#

Downloading sources & packages#

It takes some time#

Using an installed package#

What about spack load, spack unload and spack env#

What about `spack load`, `spack unload` and `spack env`#