This is a crosspost from VanessaSaurus, dinosaurs, programming, and parsnips. See the original post here.
This last week was Supercomputing 21, and I participated in the Spack BOF, where “BOF” means “Birds of a Feather.” I guess dinosaurs are birds, or bird-ish? Feathers? Anyway, if you are looking for things that I talked about, they were:
But that’s not what I want to talk about today! There was some discussion about an idea (and I cannot quote because I don’t remember the exact words) but it was something like this:
Something something… install spack packages from GitHub?
I can’t remember the details, but it prompted me to (twice!) in the chat say “You mean, like Go?” because I absolutely love that build system. Anyway, I went on my merry business that week. Until today, when this little thought bubbled into my consciousness.
Can we build and install spack packages from GitHub?
Hmm, I don’t know. Can we? With the talented @alecbcs we were able to put together to build spack containers, but in this case we would want not just a single GitHub action to build a container deployed alongside a repository, but also an action to build a spack package and upload to GitHub packages as a binary cache.
The short story is yes, these things are possible! I put together a set of GitHub actions called spack-package-action that handles the following:
Belold the power - of GitHub workflows! 🎉️ We can use GitHub composite actions to assemble a structure of folders, each with an action.yml to define one of more steps, and a folder of scripts!
# This one builds a container, uses: vsoch/spack-package-action/container@main
├── container
│ ├── action.yml
│ ├── default.yaml
│ └── scripts
│ ├── build.sh
│ ├── release.sh
│ └── set_root.sh
# This one installs spack, uses: vsoch/spack-package-action/install@main
├── install
│ ├── action.yml
│ └── scripts
│ └── install.sh
# This one builds package binaries, uses: vsoch/spack-package-action/package@main
└── package
├── action.yml
└── scripts
├── build.sh
├── release.sh
└── set_root.sh
Each folder above shows a hodge-podge of bash scripts and yaml that make this workflow work! Then, you might have a workflow that include any of these jobs:
name: Spack Package Building
on:
pull_request: []
push:
branches:
- main
jobs:
install-spack:
runs-on: ubuntu-latest
name: Install Spack
steps:
- name: Checkout
uses: actions/checkout@v2
- name: Install Spack
uses: vsoch/spack-package-action/install@main
# This builds spack binaries for the build cache for a package of choice
build-binaries:
runs-on: ubuntu-latest
permissions:
packages: write
name: Build Package Binaries
steps:
- name: Checkout
uses: actions/checkout@v2
- name: Build Spack Package
uses: vsoch/spack-package-action/package@main
with:
package: zlib
token: $
deploy: $
# This builds a spack container given a spack.yaml
build-container-spack-yaml:
runs-on: ubuntu-latest
permissions:
packages: write
name: Build Package Container spack.yaml
steps:
- name: Checkout
uses: actions/checkout@v2
- name: Build Spack Container
uses: vsoch/spack-package-action/container@main
with:
spack_yaml: spack/spack.yaml
token: $
deploy: $
# This builds a spack container for a package of choice
build-container:
runs-on: ubuntu-latest
permissions:
packages: write
name: Build Package Container
steps:
- name: Checkout
uses: actions/checkout@v2
- name: Build Spack Container
uses: vsoch/spack-package-action/container@main
with:
package: zlib
token: $
deploy: $
Here are some notes for the different actions:
This is an easy way to install spack! Along with a list of comma separated “repos” to add extra repos (GitHub urls) you can specify a branch/release, a custom root (defaults to /opt/spack) and a boolean “full_clone.” By default we clone with a depth of 1 for a faster clone, but if you need the full git history you’d want to set this to true. And the really cool thing about this step, and composite actions, is that I wound up using it in the other composite actions. I wasn’t sure if GitHub would limit me to the scope of the action root, but what I could actually do is just get the directory name (the root of the repository) for a context one level up, and then specify to use install. For example, this is within the package action.yml:
...
- name: Set Root Directory
env:
ACTION_PATH: $
run: $/scripts/set_root.sh
shell: bash
The script “set_root.sh” is very silly, it sets the root as the parent of our current directory (shortened for brevity):
#!/bin/bash
ACTION_ROOT=$(dirname $ACTION_PATH)
echo "ACTION_ROOT=${ACTION_ROOT}" >> $GITHUB_ENV
And then in the next step, we run the install script from that context, and with our variables of choice.
...
- name: Install Spack and Dependencies
env:
INPUT_BRANCH: $
INPUT_RELEASE: $
INPUT_REPOS: $
INPUT_ROOT: /opt/spack
run: $/install/scripts/install.sh
shell: bash
Pretty neat, yeah? With this approach I can use this action in the install folder in the other two folders, container and package respectively.
The package builder is essentially going to build you the same set of files you’d get by adding something to the spack build cache.
You can either give it a package name, or a local package.py file for an existing (or not existing) package.
The resulting uploaded package is going to have a name that corresponds to what we’d keep in the build cache, my idea being we
will eventually be able to query a packages endpoint to find files of this format.
If this file is discovered you’ll essentially use spack develop to install from that root. Note that I haven’t tested
this fully and should make some time soon, but please open an issue if you try it out and run into trouble!
The interestingIt’s going to have a build hash that depends on the GitHub runner to some extent, and you’ll want to add flags as an action argument to get
better control of your package target, compiler, etc (interestingly, the runners come with quite a few ready to go!).
The container builder is similar to the package action, but you can also provide a “spack_yaml” instead of a package to build a container from it with “spack containerize.” I intended to add a bunch of spack package and compiler labels, but realized this would be better done upstream (within the file, before it’s built to avoid the double build). You can also specify the different branch/release of spack to use for containerize, and a tag that will default to latest if not set.
What does this result in? On your deploy (release) triggers (when deploy is true), you’ll build and deploy packages not only for a container that has spack, but also for a binary for a spack build cache! You can take a look at the example packages here. This is done by way of the magic of oras. Let’s take a look at how we might pull an oras artifact:
$ oras pull ghcr.io/vsoch/spack-package-action/linux-ubuntu20.04-broadwell-gcc-10.3.0-zlib-1.2.11-5vlodp7yawk5elx4dfhnpzmpg743fwv3.spack:d115bcc4
Downloaded 5ee1f2ed8b80 spack-package.tar.gz
Pulled ghcr.io/vsoch/spack-package-action/linux-ubuntu20.04-broadwell-gcc-10.3.0-zlib-1.2.11-5vlodp7yawk5elx4dfhnpzmpg743fwv3.spack:d115bcc4
Digest: sha256:102901abeb89676e466184df1a87a23916febb465f688e5f4c12174263b98f9b
What did we pull?
$ ls
container install opt package README.md spack spack-package.tar.gz
Let’s look inside!
$ tar -xzvf spack-package.tar.gz
build_cache/
build_cache/linux-ubuntu20.04-broadwell/
build_cache/linux-ubuntu20.04-broadwell/gcc-10.3.0/
build_cache/linux-ubuntu20.04-broadwell/gcc-10.3.0/zlib-1.2.11/
build_cache/linux-ubuntu20.04-broadwell/gcc-10.3.0/zlib-1.2.11/linux-ubuntu20.04-broadwell-gcc-10.3.0-zlib-1.2.11-5vlodp7yawk5elx4dfhnpzmpg743fwv3.spack
build_cache/linux-ubuntu20.04-broadwell-gcc-10.3.0-zlib-1.2.11-5vlodp7yawk5elx4dfhnpzmpg743fwv3.spec.json
build_cache/_pgp/
build_cache/_pgp/03335A5FDBD232812567D91E07AA94F305E9B077.pub
Wow! So oras pull resulted in the “spack-package.tar.gz” to be in the present working directory, which has the contents of a build cache with one package. This is exciting. Let’s chat about why next.
The spack build cache is a powerful thing. When you go through the steps to create your own or connect to an existing one, it speeds up builds a lot! But historically I’ve just found this whole set up steps hard to do. If I wanted to go crazy and build a bunch of stuff and populate a cache, I’d have to give AWS some of my moolas, or use someone else’s moolahs. I’d rather not do that if I don’t need to. So this thought occurred to me earlier today:
Why not GitHub packages?
And yes, why not! As we’ve seen from the above, we are able to generate a programatically accessible build cache (artifacts) for a spack package, and store in GitHub packages. If this is an interesting or desired direction, the next step would be for us to put our heads together and decide how we might want this to work. Should we have different automated caches with different families of packages? Should we store some more permanent key alongside the repository instead of generating to sign on the fly? There are lots of cool questions to ask, so if you are interested in discussion please open an issue to discuss or ping me on Spack slack.
I’ve shared many times before that I really like autamus. How cool would it be to empower people to build spack packages, via their own repos, to produce containers for others to use? The idea that one repository == one spack package is relly neat, especially if the repository wants to have tighter control over releasing some kind of artifact with spack. One thing I wish we could do better is always testing new versions of packages, and for packages provided alongside GitHub repos, and alongside the code they provide, we could always have this testing.
So is this the only way? Of course not! I literally came up with this during an afternoon, and I’d say everything is subject to change. Here are some questions I’d like to propose to you, dear reader.