I had a complex three-axis GitHub Action matrix, but needed to skip some combinations. I couldn’t get what I needed with the direct YAML syntax, so I used Cog to generate the matrix with Python.
The matrix made Python wheels with cibuildwheel, and it worked. It had 15 jobs, but they built different numbers of architectures (ubuntu made three, windows made two, macos made only one). This made the overall run take longer, and made it harder to dig through logs to see if everything went OK. Conceptually, the matrix was three-axis, but expressed as two-axis, with a list of architectures for each job:
strategy:
matrix:
os:
- ubuntu-latest
- macos-latest
- windows-latest
cibw_build:
- cp36
- cp37
- cp38
- cp39
- cp310
include:
- os: ubuntu-latest
cibw_arch: x86_64 i686 aarch64
- os: windows-latest
cibw_arch: x86 AMD64
- os: macos-latest
cibw_arch: x86_64
I wanted to make the architectures a third axis, but couldn’t figure out how to use the YAML syntax to limit the choices for each OS. It seemed like the only way to get a ragged three-axis matrix was to list the combinations explicitly. If you know how, I’m still interested to know.
What I wanted was a way to compute the matrix with a bit more power. There are examples out there of using fromJSON to build a matrix, but I didn’t need it to be recomputed every run. I just wanted a way to not have to type out 30 combinations by hand.
I’ve often needed this sort of thing: a static file with just a bit of computed content. This is what Cog was meant for, and it worked great here too. This is what my computed matrix looks like now:
strategy:
matrix:
include:
# To change the matrix, edit the choices, then process this file with cog:
#
# $ python -m pip install cogapp
# $ python -m cogapp -rP .github/workflows/kit.yml
#
#
# [[[cog
# #----- vvv Choices for the matrix vvv -----
# oss = ["ubuntu", "macos", "windows"]
# pys = ["cp36", "cp37", "cp38", "cp39", "cp310"]
# archs = {
# "ubuntu": ["x86_64", "i686", "aarch64"],
# "macos": ["x86_64"],
# "windows": ["x86", "AMD64"],
# }
# #----- ^^^ ---------------------- ^^^ -----
#
# import json
# for the_os in oss:
# for the_py in pys:
# for the_arch in archs[the_os]:
# them = {
# "os": the_os,
# "py": the_py,
# "arch": the_arch,
# }
# print(f"- {json.dumps(them)}")
# ]]]
- {"os": "ubuntu", "py": "cp36", "arch": "x86_64"}
- {"os": "ubuntu", "py": "cp36", "arch": "i686"}
- {"os": "ubuntu", "py": "cp36", "arch": "aarch64"}
- {"os": "ubuntu", "py": "cp37", "arch": "x86_64"}
- {"os": "ubuntu", "py": "cp37", "arch": "i686"}
- {"os": "ubuntu", "py": "cp37", "arch": "aarch64"}
- {"os": "ubuntu", "py": "cp38", "arch": "x86_64"}
- {"os": "ubuntu", "py": "cp38", "arch": "i686"}
- {"os": "ubuntu", "py": "cp38", "arch": "aarch64"}
- {"os": "ubuntu", "py": "cp39", "arch": "x86_64"}
- {"os": "ubuntu", "py": "cp39", "arch": "i686"}
- {"os": "ubuntu", "py": "cp39", "arch": "aarch64"}
- {"os": "ubuntu", "py": "cp310", "arch": "x86_64"}
- {"os": "ubuntu", "py": "cp310", "arch": "i686"}
- {"os": "ubuntu", "py": "cp310", "arch": "aarch64"}
- {"os": "macos", "py": "cp36", "arch": "x86_64"}
- {"os": "macos", "py": "cp37", "arch": "x86_64"}
- {"os": "macos", "py": "cp38", "arch": "x86_64"}
- {"os": "macos", "py": "cp39", "arch": "x86_64"}
- {"os": "macos", "py": "cp310", "arch": "x86_64"}
- {"os": "windows", "py": "cp36", "arch": "x86"}
- {"os": "windows", "py": "cp36", "arch": "AMD64"}
- {"os": "windows", "py": "cp37", "arch": "x86"}
- {"os": "windows", "py": "cp37", "arch": "AMD64"}
- {"os": "windows", "py": "cp38", "arch": "x86"}
- {"os": "windows", "py": "cp38", "arch": "AMD64"}
- {"os": "windows", "py": "cp39", "arch": "x86"}
- {"os": "windows", "py": "cp39", "arch": "AMD64"}
- {"os": "windows", "py": "cp310", "arch": "x86"}
- {"os": "windows", "py": "cp310", "arch": "AMD64"}
# [[[end]]]
If you haven’t seen cog before, this is how it works: it finds chunks of
Python code between [[[cog and ]]]
markers, executes them, and inserts the output into the file up to the
[[[end]]] marker. Existing output is replaced.
Here, the 30 lines of combinations are the output. They weren’t in the file originally; they were created when I ran cog and it re-wrote the whole file. If I change the lists of choices, or the Python code, and re-run cog, it will remove those 30 lines and replace them with the new output.
This is perfect for this use: the choices for the matrix are only going to change very infrequently, and manually. When the choices need to change, I can edit the lists in the Python code, and run cog again to update the generated matrix.
Comments
I suspect that this is the way to use YAML, although I’m no Github expert and haven’t tried it.
@Karl thanks, I see what you are going for here, but I couldn’t make it work. I made a number of changes, but the problem was always the “if” clause complaining, “Unrecognized named-value: ‘matrix’ "
It seems like the matrix can’t be used in the job’s if clause?
BTW, here’s where my experiment ended up: https://github.com/nedbat/native-matrix/blob/master/.github/workflows/matrix.yml
Yes. Looks like you can’t use the “matrix context” in the job’s “if” clause: https://docs.github.com/en/actions/learn-github-actions/contexts#context-availability
You could always be brutal and inelegant and use the step’s “if” clause, repeating the same if over and over. Except for where there is already an if in the step, where things get even uglier and && is required to add your expression in. :-/ Seems not worth it.
You might be able to use the “inputs” context in the job’s “if” by creating a re-usable workflow and calling it once per OS. https://docs.github.com/en/actions/learn-github-actions/reusing-workflows
Add a comment: