Load-balanced xdist

Saturday 11 December 2021

I wrote a pytest plugin to evenly balance tests across xdist workers.

Back story: the coverage.py test suite seemed to be running oddly: it would run to near-completion, and then stall before actually finishing. To understand why, I added some debug output to see what tests were running on which workers.

I have some very slow tests (they create a virtualenv and install packages). It turned out those tests were being run near the end of the test suite, after their worker had already run a bunch of other tests. So that one worker was taking 10 seconds longer to finish than all the others. This is what made the test suite seem to stall at the end.

I figured it would be easy to schedule tests more optimally. We could record the time each test takes, then use those times in the next test run to schedule the longer tests first, and to balance the total time across workers.

The result is balance_xdist_plugin.py (commit). It’s written to be a pytest plugin, though it’s still in the coverage.py repo, so it’s not usable by others yet. And there are two things that aren’t fully general:

  • The data is written to a “tmp” directory, when it should use the pytest caching feature.
  • The number of workers is assumed to be 8, because I couldn’t figure out how to get the true number.

You can indicate that certain tests should all be assigned to the same worker. This helps with slow session-scoped fixtures, like my virtualenv-creating tests. It’s also an escape hatch if you have tests that aren’t truly isolated from each other. (Full disclosure: coverage.py has a few of these and I can’t figure out what’s wrong...)

The plugin worked: the test suite runs slightly faster than before, but as is typical, not as much faster as I thought it would. A side-benefit is that the fastest tests are now run at the end, so there’s a satisfying acceleration toward the finish line.

Maybe this plugin will be useful to others? Maybe people have improvements?

Comments

[gravatar]

Pamela McA’Nulty said:

Put 2-4 very fast tests first so you catch config/environmental problems /import-errors/etc quickly without having to wait for potentially very long running tests.

[gravatar]

Oops: contrary to my claims in the post above, the tests do not run in descending time order. I tried adding a hook for pytest_collection_modifyitems to change the order, but it wasn’t called in the main process, and didn’t seem to affect the order tests were run. More mysteries…

Add a comment:

Ignore this:
Leave this empty:
Name is required. Either email or web are required. Email won't be displayed and I won't spam you. Your web site won't be indexed by search engines.
Don't put anything here:
Leave this empty:
Comment text is Markdown.