Coverage goals

Monday 1 November 2021

There’s a feature request to add a per-file threshold to I didn’t add the feature, I wrote a proof-of-concept: has a --fail-under option that will check the total coverage percentage, and exit with a failing status if it is too low. This lets people set a goal, and then check that they are meeting it in their CI systems.

The feature request is to check each file individually, rather than the project as a whole, to exert tighter control over the goal. That sounds fine, but I could see that it would actually be more complicated than that, because people sometimes have more complicated goals: 100% coverage in tests and 85% in product code, or whatever.

I suggested implementing it as a separate tool that used data from a JSON report. Then, I did just that.

The tool is flexible: you give it a percentage number, and then a list of glob patterns. It collects up the files that match the patterns, and checks the coverage of that set of files. You can choose to measure the group as a whole, or each file individually. Patterns can be negated to remove files from consideration.

For example:

# Check all Python files collectively, except in the tests/ directory.
$ python --group 85 '**/*.py' '!tests/*.py'

# We definitely want complete coverage of anything related to html.
$ python --group 100 '**/*html*.py'

# No Python file should be below 90% covered.
$ python --file 90 '**/*.py'

Each run of checks one set of files against one goal, but you can run it multiple times if you want to check multiple goals.

If you want to have more control over your coverage goals, give a try. It might turn into a full-fledged feature, or maybe it’s enough as it is.

Feedback is welcome, either here or on the original feature request.



BTW, people have also wanted to check absolute line counts rather than percentages:

Add a comment:

Ignore this:
Leave this empty:
Name is required. Either email or web are required. Email won't be displayed and I won't spam you. Your web site won't be indexed by search engines.
Don't put anything here:
Leave this empty:
Comment text is Markdown.