Coverage.py and third-party code

Monday 12 April 2021

I’ve made a change to coverage.py, and I could use your help testing it before it’s released to the world.

tl;dr: install this and let me know if you don’t like the results:
pip install coverage==5.6b1

What’s changed? Previously, coverage.py didn’t understand about third-party code you had installed. With no options specified, it would measure and report on that code, for example in site-packages. A common solution was to use --source=. to only measure code in the current directory tree. But many people put their virtualenv in the current directory, so third-party code installed into the virtualenv would still get reported.

Now, coverage.py understands where third-party code gets installed, and won’t measure code it finds there. This should produce more useful results with less work on your part.

This was a bit tricky because the --source option can also specify an importable name instead of a directory, and it had to still measure that code even if it was installed where third-party code goes.

As of now, there is no way to change this new behavior. Third-party code is never measured.

This is kind of a big change, and there could easily be unusual arrangements that aren’t handled properly. I would like to find out about those before an official release. Try the new version and let me know what you find out:

pip install coverage==5.6b1

In particular, I would like to know if any of the code you wanted measured wasn’t measured, or if there is code being measured that “obviously” shouldn’t be. Testing on Debian (or a derivative like Ubuntu) would be helpful; I know they have different installation schemes.

If you see a problem, write up an issue. Thanks for helping.

Comments

[gravatar]
No issues with my setup of "path pointing to source, path pointing inside virtualenv (with wildcards), source pointing to source, source pointing inside virtualenv (no wildcards)". I do have the feeling it should be possible to simplify that somehow, but last time I tried, I just ended up extremely confused.
[gravatar]
@mwchase: I'm not quite sure what you meant by your setup, but I'm hoping this change to coverage.py will mean less futzing around with configuration to get the results you want.
[gravatar]
This is so cool! Thank you!!!
[gravatar]
I took another crack at simplifying things, and here's where I am now:

For context, this is in a repository that tracks multiple Python packages, some of which depend on each other. Each package is tested independently, and I combine the coverage data, for two reasons: one, general convenience of only having one report to look at, and two, I'm using the paths configuration to map virtualenv paths to source directories, and this seems to require coverage combine.

I was formerly using only paths in my source directive, and filtering out "incidental" coverage from dependent packages by specifying the virtualenv linked to a given run. This had the downside of requiring me to specify the virtualenv associated with a given run.

Right now, I've got slightly improved wildcards in my paths directive, no source directive in the coveragerc, and I'm specifying the module name, test path, and source path in the arguments I'm sending to the coverage runs.

The one thing I'd like, and I think this would have been an issue if I'd messed with this arrangement before the beta, is the ability to force a --source argument to be interpreted as a module instead of a path, because the packages usually have the same name as their top-level module; I worked around this by bundling them up in an additional top-level directory, which I'm not sure how I feel about yet.

In any case, the coverage data looks good. The changes in the beta, near as I can tell, haven't messed with anything. Everything I brought up was unrelated to the changes, oh well.
[gravatar]
@mwchase Good to hear, thanks. There's a "source_pkgs" setting in the .coveragerc file that specifies sources as package names, to deal with the ambiguity between source directory names and package names.
[gravatar]
(Very minor update. To anyone wondering if there was a good reason for specifying both the importable module and the source tree, the answer is apparently no. I just need the importable module.)
[gravatar]

Hi Ned, I’m new to the python coverage tool, sorry if my question seems basic. My use case is kinda the opposite that I want to focus coverage on only the 3rd party dependencies. I want to know how my project code is using those dependencies. My project uses Poetry, such that the 3rd party dependencies are deployed in /usr/local/…/.virtualenvs/…/lib/phython../site-packages (outside of the project folder). I tried to run the command, coverage run -m -include ‘venv/*’ pytest some_test_script_that_imports_3p.py, but it doesn’t seem to work. What are my options? So far I have succeeded in getting coverage for src, opt, python’s stdlib, but not the 3rd party.

[gravatar]

@Xiaoxuan: this won’t be a good place to get support. You can write an issue at https://github.com/nedbat/coveragepy

[gravatar]

Thank you! Posted https://github.com/nedbat/coveragepy/issues/1759. New to open source/github usage as well, still trying to find my way around.

Add a comment:

Ignore this:
Leave this empty:
Name is required. Either email or web are required. Email won't be displayed and I won't spam you. Your web site won't be indexed by search engines.
Don't put anything here:
Leave this empty:
Comment text is Markdown.