I wrote a pytest plugin to evenly balance tests across xdist workers.

Back story: the coverage.py test suite seemed to be running oddly: it would run to near-completion, and then stall before actually finishing. To understand why, I added some debug output to see what tests were running on which workers.

I have some very slow tests (they create a virtualenv and install packages). It turned out those tests were being run near the end of the test suite, after their worker had already run a bunch of other tests. So that one worker was taking 10 seconds longer to finish than all the others. This is what made the test suite seem to stall at the end.

I figured it would be easy to schedule tests more optimally. We could record the time each test takes, then use those times in the next test run to schedule the longer tests first, and to balance the total time across workers.

The result is balance_xdist_plugin.py (commit). It’s written to be a pytest plugin, though it’s still in the coverage.py repo, so it’s not usable by others yet. And there are two things that aren’t fully general:

• The data is written to a “tmp” directory, when it should use the pytest caching feature.
• The number of workers is assumed to be 8, because I couldn’t figure out how to get the true number.

You can indicate that certain tests should all be assigned to the same worker. This helps with slow session-scoped fixtures, like my virtualenv-creating tests. It’s also an escape hatch if you have tests that aren’t truly isolated from each other. (Full disclosure: coverage.py has a few of these and I can’t figure out what’s wrong...)

The plugin worked: the test suite runs slightly faster than before, but as is typical, not as much faster as I thought it would. A side-benefit is that the fastest tests are now run at the end, so there’s a satisfying acceleration toward the finish line.

Maybe this plugin will be useful to others? Maybe people have improvements?