Sunday 10 January 2016

I wasn't planning on any big features for 4.1, but I ended up making a big change. tl;dr: branch coverage is implemented differently, and as a result, your coverage totals are slightly different. Try it: 4.1 beta 1.

Because of Python 3.5's async and await keywords, the existing branch analysis based on bytecode was completely out of gas. The code had long felt jury-rigged, and there were long-standing bugs that seemed impossible to solve. The async features compiled very differently than their synchronous counterparts, and I didn't see a way to bring them into the bytecode fold.

So I ripped out the bytecode analysis and replaced it with AST (Abstract Syntax Tree) analysis. I like it much better: it's based on the structure of the code that people see and can reason about. Four old bugs were fixed as a result, along with the two or three new bugs reported on Python 3.5.

As a result though, now calculates totals differently, because the full set of possible code paths is different. So your results will likely shift a little bit, especially if you are using branch measurement. They might be higher, they might be lower. For example, class docstrings now count as executable statements (because they are executable statements), and paths through "except" clauses probably were being overlooked.

Please try it out and let me know if you see anything wrong. I won't be surprised if there are complex code structures that are analyzed wrong somehow. 4.1b1: you know you want it...


Joe Tennies 2:18 AM on 11 Jan 2016

Does this lead the way to a version of that could do MCDC? I would have some interest in that. I have to double check with my company, but I think I could help with it. I am hoping to get to PyCon this year, so I would love to sprint on that. (I do avionics work, so I'm pretty familiar.)

Ned Batchelder 11:51 AM on 11 Jan 2016

@Joe: MCDC is a huge topic. Among the challenges: 1) Python only reports on statements executed, not chunks within statements, though might be a solution; 2) How to report the results; 3) This would be a little-used feature that would add a lot of complexity, how can we balance those?

The best way to proceed is to open an issue on bitbucket, and we can flesh out some details.

Add a comment:

Ignore this:
not displayed and no spam.
Leave this empty:
not searched.
Name and either email or www are required.
Don't put anything here:
Leave this empty:
URLs auto-link and some tags are allowed: <a><b><i><p><br><pre>.