I need help with coverage.py

Monday 12 August 2013This is more than 11 years old. Be careful.

Coverage.py is kind of stuck. I haven’t done much to it in the last six months or so. Well, I’ve fixed six bugs, but I haven’t released them because I want to fix more.

In particular, I want to fix the branch coverage bugs. When I wrote the branch coverage feature, I based it on bytecode analysis of the code. Coverage.py finds the possible branches by looking at the bytecode to decide which of them can jump to which . Some of those bugs are because I don’t fully understand CPython bytecode.

To help me learn all the ins and outs of the CPython bytecode, I started byterun, a pure-Python implementation of a Python bytecode runner, or VM. Actually, I didn’t start it, I found a thing written by Paul Swartz, and renovated and extended it. It works pretty well, but isn’t complete. In particular, some of the more complex bytecodes, dealing with the block stack, don’t work properly on Python 3. I would love some help finishing byterun.

Then I need to figure out if analyzing bytecode is a reasonable way to find branches in a program. I think when I started down this path, I thought that everything would be compiled to simple bytecodes like “if cond jump to x,” but that’s not what happens. Bytecodes like WITH_CLEANUP are complex beasts with sub-optimal documentation, and they behave slightly differently in Python 2 and Python 3.

Of course I would rather not rip out the bytecode approach and implement an AST approach, but I would like to know whether that’s the way to get branch coverage that truly works well.

I’m looking for collaborators to help get through this issue. I need to get past it so I can get back to implementing features people want, like gevent support, or faster reporting, or getting rid of pickles!

Comments

[gravatar]
The peephole optimizer is likely to cause you endless pain with bytecode based analysis - it can make branches in the source code disappear from the bytecode...
[gravatar]
@Nick: indeed, it does: see bug #198: Continue marked as not covered. This is why I believe CPython should have a way to completely disable the peephole optimizer. Comment on the CPython ticket if you agree!
[gravatar]
Ha, so this is why coverage sometimes tells me some "branches" are not covered, even though I can't see an actual branch :-)

Also, I would like to point out that "branch coverage" is a fuzzy thing. You point out the context manager example: if a "with" block always ends up successfully, should it be counted as missing the exception case? How about try..finally pairs?

I think "branch coverage" may therefore be misguided and the only reasonably measurable thing is actualy code coverage (i.e. coverage of bytecode blocks).
[gravatar]
@Antoine: you are right, originally branch coverage was only defined for the classic control structures: if, while, etc. Exceptions throw a wrench in the works. In fact, in Python, any statement can throw an exception, so should we measure if each one did? Of course not, since many in fact never will.

I would still like to improve the feature where I can!
[gravatar]
if possible,I would like to volunteer work on this project. if it is require. it will be great opportunity for me to learning thing

Add a comment:

Ignore this:
Leave this empty:
Name is required. Either email or web are required. Email won't be displayed and I won't spam you. Your web site won't be indexed by search engines.
Don't put anything here:
Leave this empty:
Comment text is Markdown.