Coverage.py and third-party code

Monday 12 April 2021

I’ve made a change to coverage.py, and I could use your help testing it before it’s released to the world.

tl;dr: install this and let me know if you don’t like the results:
pip install coverage==5.6b1

What’s changed? Previously, coverage.py didn’t understand about third-party code you had installed. With no options specified, it would measure and report on that code, for example in site-packages. A common solution was to use --source=. to only measure code in the current directory tree. But many people put their virtualenv in the current directory, so third-party code installed into the virtualenv would still get reported.

Now, coverage.py understands where third-party code gets installed, and won’t measure code it finds there. This should produce more useful results with less work on your part.

This was a bit tricky because the --source option can also specify an importable name instead of a directory, and it had to still measure that code even if it was installed where third-party code goes.

As of now, there is no way to change this new behavior. Third-party code is never measured.

This is kind of a big change, and there could easily be unusual arrangements that aren’t handled properly. I would like to find out about those before an official release. Try the new version and let me know what you find out:

pip install coverage==5.6b1

In particular, I would like to know if any of the code you wanted measured wasn’t measured, or if there is code being measured that “obviously” shouldn’t be. Testing on Debian (or a derivative like Ubuntu) would be helpful; I know they have different installation schemes.

If you see a problem, write up an issue. Thanks for helping.

Comments

[gravatar]

No issues with my setup of "path pointing to source, path pointing inside virtualenv (with wildcards), source pointing to source, source pointing inside virtualenv (no wildcards)". I do have the feeling it should be possible to simplify that somehow, but last time I tried, I just ended up extremely confused.

[gravatar]

@mwchase: I'm not quite sure what you meant by your setup, but I'm hoping this change to coverage.py will mean less futzing around with configuration to get the results you want.

[gravatar]

This is so cool! Thank you!!!

[gravatar]

I took another crack at simplifying things, and here's where I am now:

For context, this is in a repository that tracks multiple Python packages, some of which depend on each other. Each package is tested independently, and I combine the coverage data, for two reasons: one, general convenience of only having one report to look at, and two, I'm using the paths configuration to map virtualenv paths to source directories, and this seems to require coverage combine.

I was formerly using only paths in my source directive, and filtering out "incidental" coverage from dependent packages by specifying the virtualenv linked to a given run. This had the downside of requiring me to specify the virtualenv associated with a given run.

Right now, I've got slightly improved wildcards in my paths directive, no source directive in the coveragerc, and I'm specifying the module name, test path, and source path in the arguments I'm sending to the coverage runs.

The one thing I'd like, and I think this would have been an issue if I'd messed with this arrangement before the beta, is the ability to force a --source argument to be interpreted as a module instead of a path, because the packages usually have the same name as their top-level module; I worked around this by bundling them up in an additional top-level directory, which I'm not sure how I feel about yet.

In any case, the coverage data looks good. The changes in the beta, near as I can tell, haven't messed with anything. Everything I brought up was unrelated to the changes, oh well.

[gravatar]

@mwchase Good to hear, thanks. There's a "source_pkgs" setting in the .coveragerc file that specifies sources as package names, to deal with the ambiguity between source directory names and package names.

[gravatar]

(Very minor update. To anyone wondering if there was a good reason for specifying both the importable module and the source tree, the answer is apparently no. I just need the importable module.)

Add a comment:

Ignore this:
Leave this empty:
Name is required. Either email or web are required. Email won't be displayed and I won't spam you. Your web site won't be indexed by search engines.
Don't put anything here:
Leave this empty:
URLs auto-link and some tags are allowed: <a><b><i><p><br><pre>.