Ned Batchelder : Blog | Code | Text | Site
Statement coverage for Python: design and analysis
» Home : Code : Python Modules : coverage
This page is orignally from http://www.garethrees.org/2001/12/04/python-coverage/design.html, which seems to be defunct. I grabbed the text from archive.org, and present it here for posterity. I've updated the links to point to appropriate pages on this site.
This document lists the requirements for a statement coverage tool
for Python, describes some issues in design and implementation, and
coverage.py with other statement coverage
You can run many tests and perform coverage analysis based on all the tests.
You can get a summary report showing coverage for a set of modules and the total.
You can annotate Python source code to show which statements are covered.
The coverage analysis deals correctly with all Python features.
You can do coverage analysis while testing interactively.
Recording of coverage information doesn't slow down a test case more than necessary.
Requirement 1 means that coverage information needs to be accumulated in a file during a sequence of tests. See [GDR 2001-12-04, 2] for the command-line interface which achieves this.
Requirement 5 means that there needs to be a documented programmatic interface. See [GDR 2001-12-04, 3] for the documented interface.
To meet requirement 4 we need to know which source lines represent
statements. Looking for non-blank, non-comment lines isn't good enough
because some statements extend across many lines, but only the first
line will appear in Python as a source line number. So we use the
Python parser [van Rossum
2001-07-20, 17.1] to parse the module sources and walk the source
tree looking for the first line of each statement (when
symbol.stmt we descend
tree until we get to a
terminal token, whose line we record).
Code on the second and subsequent lines of multi-line simple
statements is reported by the tracing interface as appearing on the
first line of the statement, so recording the first line of each
statement captures all the executed lines except for
finally lines in compound
statements. So we record the lines with these tokens separately.
No execution takes place on a line containing only the
else token of
try compound statements, so we don't record lines
else. But when we annotate a listing, such
a line should be marked as covered if and only if the following
statement is covered. So the
annotate() method has special
logic for this case.
The filename in a Python code object rarely matches the
__file__ attribute of the module to which the code belongs.
There are three kinds of difference:
module.__file__ is the compiled byte code
.pyc), but the filename for the code is the source file.
We work around this by turning
when we find it.
The file names differ as to directory, for example we may have
module.__file__ == '/dev/project/foo/module.py', but
code.co_filename == 'module.py'. We work around this by
sys.path to find the file.
The code might have been compiled somewhere temporary, for
module.__file__ == '/usr/lib/python1.5/getopt.py'
'/var/tmp/python/usr/lib/python1.5/getopt.py'. We work around
this as a last resort by stripping the directory part and then looking
for the file in
Both have these problems:
No summary report (requirement 2).
Annotations aren't accurate (requirement 4):
Second and subsequent lines of multi-line statements are marked as not executed.
Lines containing only
else: are marked as not
Comment lines and blank lines are incorrectly recognized and marked as blank if they appear in multi-line strings.
trace.py also has these problems:
Complicated programmatic interface (requirement 5).
Substantially slower than other coverage testing tools (requirement 6): see table 1.
|Test||Execution time (s)|
|[Csillag 1999-07-01]||"pycover 0.2"; Andrew Csillag; 1999-07-01; <http://www.geocities.com/drew_csillag/pycover.html>.|
|[Dalke 1999]||"trace.py"; Andrew Dalke; 1999; <ftp://starship.python.net/pub/crew/dalke/trace.py>.|
|[GDR 2001-12-04]||"Statement coverage for Python"; Gareth Rees; Ravenbrook Limited; 2001-12-04; <http://www.garethrees.org/2001/12/04/python-coverage/>|
|[van Rossum 2001-07-20]||"Python Reference Manual (release 2.1.1)"; Guido van Rossum; 2001-07-20; <http://www.python.org/doc/2.1.1/lib/lib.html>.|
Copyright © 2001 Gareth Rees. This document is provided "as is", without any express or implied warranty. In no event will the authors be held liable for any damages arising from the use of this document. You may make and distribute verbatim copies of this document provided that you do not charge a fee for this document or for its distribution.