This page is orignally from, which seems to be defunct. I grabbed the text from, and present it here for posterity. I've updated the links to point to appropriate pages on this site. / Software / Coverage

Statement coverage for Python: design and analysis

Gareth Rees, Ravenbrook Limited, 2001-12-04

1. Introduction

This document lists the requirements for a statement coverage tool for Python, describes some issues in design and implementation, and compares with other statement coverage implementations.

2. Requirements

  1. You can run many tests and perform coverage analysis based on all the tests.

  2. You can get a summary report showing coverage for a set of modules and the total.

  3. You can annotate Python source code to show which statements are covered.

  4. The coverage analysis deals correctly with all Python features.

  5. You can do coverage analysis while testing interactively.

  6. Recording of coverage information doesn't slow down a test case more than necessary.

3. Design

3.1. Interfaces

Requirement 1 means that coverage information needs to be accumulated in a file during a sequence of tests. See [GDR 2001-12-04, 2] for the command-line interface which achieves this.

Requirement 5 means that there needs to be a documented programmatic interface. See [GDR 2001-12-04, 3] for the documented interface.

3.2. Python language features

To meet requirement 4 we need to know which source lines represent statements. Looking for non-blank, non-comment lines isn't good enough because some statements extend across many lines, but only the first line will appear in Python as a source line number. So we use the Python parser [van Rossum 2001-07-20, 17.1] to parse the module sources and walk the source tree looking for the first line of each statement (when tree[0] == symbol.stmt we descend tree[1] until we get to a terminal token, whose line we record).

Code on the second and subsequent lines of multi-line simple statements is reported by the tracing interface as appearing on the first line of the statement, so recording the first line of each statement captures all the executed lines except for elif, except and finally lines in compound statements. So we record the lines with these tokens separately.

See the find_statements() method.

No execution takes place on a line containing only the else token of if, while and try compound statements, so we don't record lines containing only else. But when we annotate a listing, such a line should be marked as covered if and only if the following statement is covered. So the annotate() method has special logic for this case.

3.3. Filenames

The filename in a Python code object rarely matches the __file__ attribute of the module to which the code belongs. There are three kinds of difference:

  1. module.__file__ is the compiled byte code (.pyc), but the filename for the code is the source file. We work around this by turning .pyc to .py when we find it.

  2. The file names differ as to directory, for example we may have module.__file__ == '/dev/project/foo/', but code.co_filename == ''. We work around this by search sys.path to find the file.

  3. The code might have been compiled somewhere temporary, for example module.__file__ == '/usr/lib/python1.5/' but code.co_filename == '/var/tmp/python/usr/lib/python1.5/'. We work around this as a last resort by stripping the directory part and then looking for the file in sys.path.

See the canonical_filename() method.

4. Comparison with other implementations

I discovered two other statement coverage testing tools for Python. Both [Dalke 1999] and pycover 0.2 [Csillag 1999-07-01]. Neither met all my requirements.

Both have these problems: also has these problems:

Table 1. Performance of coverage implementations

Test Execution time (s)
No coverage
10 × 15 59 120 65 21 180 306 184

A. References

[Csillag 1999-07-01] "pycover 0.2"; Andrew Csillag; 1999-07-01; <>.
[Dalke 1999] ""; Andrew Dalke; 1999; <>.
[GDR 2001-12-04] "Statement coverage for Python"; Gareth Rees; Ravenbrook Limited; 2001-12-04; <>
[van Rossum 2001-07-20] "Python Reference Manual (release 2.1.1)"; Guido van Rossum; 2001-07-20; <>.

B. Document History

2001-12-04 GDR Created.

Copyright © 2001 Gareth Rees. This document is provided "as is", without any express or implied warranty. In no event will the authors be held liable for any damages arising from the use of this document. You may make and distribute verbatim copies of this document provided that you do not charge a fee for this document or for its distribution.

$Id: // $ / Software / Coverage