Uptodate

Posted by Vanessasaurus on September 19, 2021 · 16 mins read

This is a crosspost from VanessaSaurus, dinosaurs, programming, and parsnips. See the original post here.

I recently had an itch to scratch - and that itch was writing a library in Go. We don’t use Go much for my work, so I figured out a compelling reason to start a new personal project - a command line tool written in Go (and matching GitHub action) to help keep things up to date in a repository. Appropriately, I called it uptodate! It was hugely inspired from the binoc (short for “binoculars”) library that can also perform specific kinds of updates, but I wanted more of a focus on Docker, and to have total control so I could go wild and crazy with writing Go code without worrying about forcing it on the owner, alecbcs, to merge my wild ideas.


Uptodate

Uptodate is a command line tool in Go and GitHub action that makes it easy to:

  1. Update FROM statements in Dockerfile to have the latest shas
  2. Update build arguments that are for spack versions, GitHub releases and commits, and container hashes.
  3. Generate a matrix of Docker builds from a single configuration file
  4. Generate a matrix of changed files in a repository.
  5. List Dockerfile in a repository that have been changed.

With all of the above, you can imagine a workflow that first updates Dockerfile FROM statements and build args, and then re-builds and deploys these containers - the assumption being that the underlying dependency such as a GitHub commit or spack version has an update. Uptodate also will take a nested structure that I call a docker “build hierarchy” and add new folders and Dockerfile when a new tag is detected. A kind of updater in uptodate is naturally called an “updater” and this means for the docker build and docker hierarchy updaters, we can write a yaml configuration file with our preferences for versions to be added, and other metadata. You should check out the user guide for detailed usage, or read about the GitHub action

How does it work?

I’ll give a brief overview of a few of the commands and then a quick example GitHub workflow, and I’ll recommend that you read the documentation for the latest updates on uptodate, harharhar. The examples below assumed that you’ve installed uptodate and have the binary “uptodate” in your path.

Dockerfile

If you have one or more Dockerfile in your repository you can run uptodate to update digests. For example:

$ uptodate dockerfile .

will find Dockerfile in the present working directory and subfolders and update. For digests, you might see that:

FROM ubuntu:20.04

is updated to

FROM ubuntu:18.04@sha256:9bc830af2bef73276515a29aa896eedfa7bdf4bdbc5c1063b4c457a4bbb8cd79

Note in the above we still have the digest and the tag, so subsequent updates can further update the sha by looking up the container based on the tag. And we can also update build arguments that match a particular format! This one, specifically:

ARG uptodate_<build-arg-type>_<build-arg-value>=<default>

The above flags the build argument for uptodate to look at using the prefix of the library name, and then the next string after the underscore is the kind of update, followed by specific metadata for that updater, and of course the value! A few examples are provided below.

Spack Build Arguments

Spack is a package manager intended for HPC, and it’s huge at the lab where I work. So naturally, it made sense for uptodate to be able to look up the latest spack versions for some package. To create an argument that matched to a spack package (and its version) you might see:

ARG uptodate_spack_ace=6.5.6

After the updater runs, if it finds a new version 6.5.12, the line will read:

ARG uptodate_spack_ace=6.5.12

This works by using the static API that is deployed alongside the Spack Packages repository that I designed earlier this year. So the updater will get the latest versions as known within the last 24 hours.

GitHub Release Build Argument

If we want an updated version from a GitHub release (let’s say the spack software itself) we might see this:

ARG uptodate_github_release_spack__spack=v0.16.1

The above will look for new releases from spack on GitHub and update as follows:

ARG uptodate_github_release_spack__spack=v0.16.2

GitHub Commit Build Argument

Similarity, if we want more “bleeding edge” changes we can ask for a commit from a specific branch, following this pattern:

ARG uptodate_github_commit_<org>__<name>__<branch>=<release-tag>

Here is an example of asking for updates for the develop branch.

ARG uptodate_github_commit_spack__spack__develop=NA

which wouldn’t care about the first “commit” NA as it would update to:

ARG uptodate_github_commit_spack__spack__develop=be8e52fbbec8106150680fc628dc72e69e5a20be

And then to use it in your Dockerfile, you might pop into an environment variable:

ENV spack_commit=${uptodate_github_commit_spack__spack__develop}

See the docs for more detailed usage and an example for the Dockerfile updater.

Docker Build

The second updater that I think is pretty useful is the Docker build updater. This updated will read a config file, an uptodate.yaml, and then follow instructions for version regular expressoins and different kinds of builds args to generate a matrix of builds (intended for GitHub actions). For example, let’s say that we start with this configuration file:


dockerbuild:
  build_args:

    # This is an example of a manual build arg, versions are required
    llvm_version:

      # The key is a shorthand used for naming (required)
      key: llvm
      versions:
       - "4.0.0"
       - "5.0.1"
       - "6.0.0"

    # This is an example of a spack build arg, the name is the package
    abyss_version:
      key: abyss
      name: abyss
      type: spack

    # This will be parsed by the Dockerfile parser, name is the container name
    ubuntu_version:

      key: ubuntu
      name: ubuntu
      type: container
      startat: "16.04"
      endat: "20.04"
      filter: 
        - "^[0-9]+[.]04$" 
      skips:
      - "17.04"
      - "19.04"

You’ll see the primary section of interest is under “dockerbuild” and under this we have three build args for a manually defined set of versions, a version from a spack package, and a container. You could run this in a repository root to look for these config files (and a Dockerfile that they render with in the same directory or below it) to generate a build matrix.

$ uptodate dockerbuild

Or to only include changed uptodate.yaml files:

$ uptodate dockerbuild --changes

If you provide a registry URI that the containers build to, we can actually check these containers to look at current build args (that are saved as labels and then viewable in the image config by uptodate) to determine if an update is needed.

$ uptodate dockerbuild --registry ghcr.io/rse-radiuss

the container. I think this is one of the neatest features - it was just added in evenings this last week! Check out an example image config that has these labels! This registry URI will also be included in the output to make it easy to build In a GitHub action, it might be used like this:

jobs:
  generate:
    name: Generate Build Matrix
    runs-on: ubuntu-latest
    outputs:
      dockerbuild_matrix: $
      empty_matrix: $

    steps:
    - uses: actions/checkout@v2
      if: github.event_name == 'pull_request'
      with:
         fetch-depth: 0
         ref: $

    - uses: actions/checkout@v2
      if: github.event_name != 'pull_request'
      with:
         fetch-depth: 0

    - name: Generate Build Matrix
      uses: vsoch/uptodate@main
      id: dockerbuild
      with: 
        root: .
        parser: dockerbuild
        flags: "--registry ghcr.io/myreponame"

    - name: View and Check Build Matrix Result
      env:
        result: $
      run: |
        echo ${result}

  build:
    needs:
      - generate
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        result: $
    if: $

    name: "Build $"
    steps:
    - name: Checkout Repository
      uses: actions/checkout@v2

    - name: Set up Docker Buildx
      uses: docker/setup-buildx-action@v1

    - name: Build $
      id: builder
      env:
        container: $
        prefix: $
        filename: $
      run: |
        basedir=$(dirname $filename)
        cd $basedir
        ${prefix} -t ${container} .

Of course you’d want to login to a registry, and then also possibly calculate metrics for the container, so consider this a very simple example. The build matrix that is being passed between those steps has entries like this:

[
  {
    "name": "ubuntu/clang/uptodate.yaml",
    "container_name": "ghcr.io/rse-radiuss/clang-ubuntu-20.04:llvm-10.0.0",
    "filename": "ubuntu/clang/Dockerfile",
    "parser": "dockerbuild",
    "buildargs": {
      "llvm_version": "10.0.0",
      "ubuntu_version": "20.04"
    },
    "command_prefix": "docker build -f Dockerfile --build-arg llvm_version=10.0.0 --build-arg ubuntu_version=20.04",
    "description": "ubuntu/clang llvm_version:10.0.0 ubuntu_version:20.04"
  },
  ...
]

Git Updater

I also like this updater because it easily generates for you a matrix of files that are changed, according to git. Running locally it looks like this:

$ ./uptodate git /path/to/repo
              _            _       _       
  _   _ _ __ | |_ ___   __| | __ _| |_ ___ 
 | | | | '_ \| __/ _ \ / _  |/ _  | __/ _ \
 | |_| | |_) | || (_) | (_| | (_| | ||  __/
  \__,_| .__/ \__\___/ \__,_|\__,_|\__\___|
       |_|                          git


  ⭐️ Changed Files ⭐️
    .github/workflows/build-matrices.yaml: Modify

And would generate a matrix for a GitHub action too:

[
  {
    "name": "Modify",
    "filename": "cli/dockerbuild.go"
  },
  {
    "name": "Modify",
    "filename": "parsers/common.go"
  },
  {
    "name": "Insert",
    "filename": "parsers/docker/buildargs.go"
  },
  {
    "name": "Modify",
    "filename": "parsers/docker/docker.go"
  },
  {
    "name": "Modify",
    "filename": "tests/ubuntu/21.04/Dockerfile"
  },
  {
    "name": "Modify",
    "filename": "tests/ubuntu/clang/Dockerfile"
  }
]

And of course you can change the default “main” to another branch:

$ ./uptodate git /path/to/repo --branch master

and that also pipes into a GitHub action. I don’t want to redundantly reproduce the docs, so if you are interested you can read more at the user guide or GitHub action pages. Mind you that the library is heavily under develop, so if you have a request for a new updater or want to report a a bug, please let me know!.

Overview

I have loved working on this library. I think it’s the first library in Go where I’ve been proficient enough to not look everything up that I need - the code has just flowed from my fingers! Mind you I’m still figuring out my own design preferences, and I’m at the stage where I’ll write a new functionality, and then immediately not like my design, and want to re-write it. But I think that means I’ll eventually get better. But it’s always good to have one or more projects you are passionate about, because I don’t personally see a point in being a software engineer if I don’t (yes, I know it makes a salary, but I require more than that).

← Previous Post Next Post →