Coverage.py v3.2 beta 1 is available, and it's got a big new feature: branch coverage. It's been a long time coming, but I'm pretty pleased with the results. I'm very interested to hear whether this is useful, and what could be improved.

Branch coverage

Coverage.py now supports branch coverage measurement. Where a line in your program could jump to more than one next line, coverage.py tracks which of those destinations are actually visited, and flags lines that haven't visited all of their possible destinations.

For example:

def my_partial_fn(x):       # line 1
    if x:                   #      2
        y = 10              #      3
    return y                #      4
    
my_partial_fn(1)

In this code, the if on line 2 could branch to either line 3 or line 4. Statement coverage would show all lines of the function as executed. But the if is always true, so line 2 never jumps to line 4. Even though line 4 is executed, coverage.py knows that it was never because of a branch from line 2.

Branch coverage would flag this code as not fully covered because of the missing jump from line 2 to line 4.

How to measure branch coverage

To measure branch coverage, run coverage.py with the --branch flag:

coverage run --branch myprog.py

When you report on the results with "coverage report" or "coverage html", the percentage of branch possibilities taken will be included in the percentage covered total for each file. The coverage percentage for a file is the actual executions divided by the execution opportunities. Each line in the file is an execution opportunity, as is each branch destination.

Currently, only HTML reports give information about which lines had missing branches. Lines that were missing some branches are shown in yellow, with an annotation at the far right showing branch destination line numbers that were not exercised.

How it works

When measuring branches, coverage.py collects pairs of line numbers, a source and destination for each transition from one line to another. Static analysis of the compiled bytecode provides a list of possible transitions. Comparing the measured to the possible indicates missing branches.

The idea of tracking how lines follow each other was from C. Titus Brown. Thanks, Titus!

Problems

Some Python constructs are difficult to measure properly. For example, an infinite loop will be marked as partially executed:

while True:                         # line 1
    if some_condition():            #      2
        break                       
    body_of_loop()                  #      4

keep_working()                      #      6

Because the loop never terminates naturally (jumping from line 1 to 6), coverage.py thinks the branch is partially executed.

Currently, if you exclude code from coverage testing, a branch into that code will still be considered executable, and may result in the branch being flagged.

A few other unexpected cases are described in branches.html, which also shows how partial branches are displayed in the HTML report.

The only way currently to initiate branch coverage is with the command-line interface. In particular, the nose coverage plugin has no way to use it.

Other work

One interesting side effect of tracking line transitions: we know where some exceptions happened because a transition happens that wasn't predicted by the static analysis. Currently, I'm not doing anything with this information. Any ideas?

tagged: , » 7 reactions

Comments

[gravatar]
Jorge Vargas 9:55 PM on 10 Nov 2009

Wow Thank you Thank you Thank you!

[gravatar]
matt harrison 11:18 PM on 10 Nov 2009

Thanks Ned (and Titus) - This is great news

[gravatar]
Andre 7:00 AM on 13 Nov 2009

I do not think that except clauses create a three-way-branch: The throw clause creates a branch into the next available except handler (or, failing that, out of the program).

[gravatar]
Magnus 2:16 AM on 27 Nov 2009

Very nice indeed!
I'm looking forward to a possible API-implementation in the future. :)

[gravatar]
Kurt McKee 10:30 PM on 28 Nov 2009

Thanks for one of the best tools I have in my toolkit. Coverage is fantastic.

[gravatar]
Uwe Lange 11:42 AM on 3 Dec 2009

Would it be possible to treat 'while True:' and 'if __name__ == "__main__":' as special cases so that they are not reported as partially executed? Otherwise a great tool: I already found several branches which were not covered by tests in my source code. coverage.py should make it into the standard library.

[gravatar]
Ned Batchelder 12:39 PM on 3 Dec 2009

@Uwe: I have to figure out what to do about "while True:", that is a real problem. The '__name__ == "__main__"' issue can already be dealt with by excluding that line from coverage reporting, either with an explicit pragma comment, or by matching it with an exclude regex.

Add a comment:

name
email
Ignore this:
not displayed and no spam.
Leave this empty:
www
not searched.
 
Name and either email or www are required.
Don't put anything here:
Leave this empty:
URLs auto-link and some tags are allowed: <a><b><i><p><br><pre>.