Computing a GitHub Action matrix with cog

Sunday 7 November 2021

I had a complex three-axis GitHub Action matrix, but needed to skip some combinations. I couldn’t get what I needed with the direct YAML syntax, so I used Cog to generate the matrix with Python.

The matrix made Python wheels with cibuildwheel, and it worked. It had 15 jobs, but they built different numbers of architectures (ubuntu made three, windows made two, macos made only one). This made the overall run take longer, and made it harder to dig through logs to see if everything went OK. Conceptually, the matrix was three-axis, but expressed as two-axis, with a list of architectures for each job:

      - ubuntu-latest
      - macos-latest
      - windows-latest
      - cp36
      - cp37
      - cp38
      - cp39
      - cp310
      - os: ubuntu-latest
        cibw_arch: x86_64 i686 aarch64
      - os: windows-latest
        cibw_arch: x86 AMD64
      - os: macos-latest
        cibw_arch: x86_64

I wanted to make the architectures a third axis, but couldn’t figure out how to use the YAML syntax to limit the choices for each OS. It seemed like the only way to get a ragged three-axis matrix was to list the combinations explicitly. If you know how, I’m still interested to know.

What I wanted was a way to compute the matrix with a bit more power. There are examples out there of using fromJSON to build a matrix, but I didn’t need it to be recomputed every run. I just wanted a way to not have to type out 30 combinations by hand.

I’ve often needed this sort of thing: a static file with just a bit of computed content. This is what Cog was meant for, and it worked great here too. This is what my computed matrix looks like now:

      # To change the matrix, edit the choices, then process this file with cog:
      # $ python -m pip install cogapp
      # $ python -m cogapp -rP .github/workflows/kit.yml
      # [[[cog
      #   #----- vvv Choices for the matrix vvv -----
      #   oss = ["ubuntu", "macos", "windows"]
      #   pys = ["cp36", "cp37", "cp38", "cp39", "cp310"]
      #   archs = {
      #       "ubuntu": ["x86_64", "i686", "aarch64"],
      #       "macos": ["x86_64"],
      #       "windows": ["x86", "AMD64"],
      #   }
      #   #----- ^^^ ---------------------- ^^^ -----
      #   import json
      #   for the_os in oss:
      #       for the_py in pys:
      #           for the_arch in archs[the_os]:
      #               them = {
      #                   "os": the_os,
      #                   "py": the_py,
      #                   "arch": the_arch,
      #               }
      #               print(f"- {json.dumps(them)}")
      # ]]]
      - {"os": "ubuntu", "py": "cp36", "arch": "x86_64"}
      - {"os": "ubuntu", "py": "cp36", "arch": "i686"}
      - {"os": "ubuntu", "py": "cp36", "arch": "aarch64"}
      - {"os": "ubuntu", "py": "cp37", "arch": "x86_64"}
      - {"os": "ubuntu", "py": "cp37", "arch": "i686"}
      - {"os": "ubuntu", "py": "cp37", "arch": "aarch64"}
      - {"os": "ubuntu", "py": "cp38", "arch": "x86_64"}
      - {"os": "ubuntu", "py": "cp38", "arch": "i686"}
      - {"os": "ubuntu", "py": "cp38", "arch": "aarch64"}
      - {"os": "ubuntu", "py": "cp39", "arch": "x86_64"}
      - {"os": "ubuntu", "py": "cp39", "arch": "i686"}
      - {"os": "ubuntu", "py": "cp39", "arch": "aarch64"}
      - {"os": "ubuntu", "py": "cp310", "arch": "x86_64"}
      - {"os": "ubuntu", "py": "cp310", "arch": "i686"}
      - {"os": "ubuntu", "py": "cp310", "arch": "aarch64"}
      - {"os": "macos", "py": "cp36", "arch": "x86_64"}
      - {"os": "macos", "py": "cp37", "arch": "x86_64"}
      - {"os": "macos", "py": "cp38", "arch": "x86_64"}
      - {"os": "macos", "py": "cp39", "arch": "x86_64"}
      - {"os": "macos", "py": "cp310", "arch": "x86_64"}
      - {"os": "windows", "py": "cp36", "arch": "x86"}
      - {"os": "windows", "py": "cp36", "arch": "AMD64"}
      - {"os": "windows", "py": "cp37", "arch": "x86"}
      - {"os": "windows", "py": "cp37", "arch": "AMD64"}
      - {"os": "windows", "py": "cp38", "arch": "x86"}
      - {"os": "windows", "py": "cp38", "arch": "AMD64"}
      - {"os": "windows", "py": "cp39", "arch": "x86"}
      - {"os": "windows", "py": "cp39", "arch": "AMD64"}
      - {"os": "windows", "py": "cp310", "arch": "x86"}
      - {"os": "windows", "py": "cp310", "arch": "AMD64"}
    # [[[end]]]

If you haven’t seen cog before, this is how it works: it finds chunks of Python code between [[[cog and ]]] markers, executes them, and inserts the output into the file up to the [[[end]]] marker. Existing output is replaced.

Here, the 30 lines of combinations are the output. They weren’t in the file originally; they were created when I ran cog and it re-wrote the whole file. If I change the lists of choices, or the Python code, and re-run cog, it will remove those 30 lines and replace them with the new output.

This is perfect for this use: the choices for the matrix are only going to change very infrequently, and manually. When the choices need to change, I can edit the lists in the Python code, and run cog again to update the generated matrix.



I suspect that this is the way to use YAML, although I’m no Github expert and haven’t tried it.

# A github action multi-dimentional build matrix

  # Run the job only when we want the given arch on the given OS
  if: contains(matrix.os.archs, matrix.arch)
    # Note references to within matrix.os' data structure
    name: >
      Build ${{ }} ${{ }} ${{ matrix.arch }}
    runs-on: ${{ matrix.os.version }}
        # We are limited to a matrix with 256 elements.
        # More than that and you can fall back to generating sparse
        # matrix elements with "include:", using cog.
          - ubuntu:
              # the key of the mapping does not seem to be made available
              # by the github API, so provide a value with the os name
              name: ubuntu
              version: ubuntu-latest
                - x86_64
                - i686
                - aarch64
          - windows:
              name: windows
              version: windows-latest
                - x86
                - amd64
          - macos:
              name: macos
              version: macos-latest
                - x86_64
          - cp36
          - cp37
          - cp38
          - cp39
          - cp310
          - x86_64
          - i686
          - aarch64
          - x86
    fail-fast: false

    - name: Setup QEMU
      # Note reference to within matrix.os' data structure
      if: == 'ubuntu'
      uses: docker/setup-qemu-action@v1
        platforms: arm64

@Karl thanks, I see what you are going for here, but I couldn’t make it work. I made a number of changes, but the problem was always the “if” clause complaining, “Unrecognized named-value: ‘matrix’ "

It seems like the matrix can’t be used in the job’s if clause?


Yes. Looks like you can’t use the “matrix context” in the job’s “if” clause:

You could always be brutal and inelegant and use the step’s “if” clause, repeating the same if over and over. Except for where there is already an if in the step, where things get even uglier and && is required to add your expression in. :-/ Seems not worth it.

You might be able to use the “inputs” context in the job’s “if” by creating a re-usable workflow and calling it once per OS.

Add a comment:

Ignore this:
Leave this empty:
Name is required. Either email or web are required. Email won't be displayed and I won't spam you. Your web site won't be indexed by search engines.
Don't put anything here:
Leave this empty:
Comment text is Markdown.