1. Introduction

This document lists the requirements for a statement coverage tool for Python, describes some issues in design and implementation, and compares coverage.py with other statement coverage implementations.

2. Requirements

1. You can run many tests and perform coverage analysis based on all the tests.

2. You can get a summary report showing coverage for a set of modules and the total.

3. You can annotate Python source code to show which statements are covered.

4. The coverage analysis deals correctly with all Python features.

5. You can do coverage analysis while testing interactively.

6. Recording of coverage information doesn't slow down a test case more than necessary.

3. Design

3.1. Interfaces

Requirement 1 means that coverage information needs to be accumulated in a file during a sequence of tests. See [GDR 2001-12-04, 2] for the command-line interface which achieves this.

Requirement 5 means that there needs to be a documented programmatic interface. See [GDR 2001-12-04, 3] for the documented interface.

3.2. Python language features

To meet requirement 4 we need to know which source lines represent statements. Looking for non-blank, non-comment lines isn't good enough because some statements extend across many lines, but only the first line will appear in Python as a source line number. So we use the Python parser [van Rossum 2001-07-20, 17.1] to parse the module sources and walk the source tree looking for the first line of each statement (when tree[0] == symbol.stmt we descend tree[1] until we get to a terminal token, whose line we record).

Code on the second and subsequent lines of multi-line simple statements is reported by the tracing interface as appearing on the first line of the statement, so recording the first line of each statement captures all the executed lines except for elif, except and finally lines in compound statements. So we record the lines with these tokens separately.

See the find_statements() method.

No execution takes place on a line containing only the else token of if, while and try compound statements, so we don't record lines containing only else. But when we annotate a listing, such a line should be marked as covered if and only if the following statement is covered. So the annotate() method has special logic for this case.

3.3. Filenames

The filename in a Python code object rarely matches the __file__ attribute of the module to which the code belongs. There are three kinds of difference:

1. module.__file__ is the compiled byte code (.pyc), but the filename for the code is the source file. We work around this by turning .pyc to .py when we find it.

2. The file names differ as to directory, for example we may have module.__file__ == '/dev/project/foo/module.py', but code.co_filename == 'module.py'. We work around this by search sys.path to find the file.

3. The code might have been compiled somewhere temporary, for example module.__file__ == '/usr/lib/python1.5/getopt.py' but code.co_filename == '/var/tmp/python/usr/lib/python1.5/getopt.py'. We work around this as a last resort by stripping the directory part and then looking for the file in sys.path.

See the canonical_filename() method.

4. Comparison with other implementations

I discovered two other statement coverage testing tools for Python. Both trace.py [Dalke 1999] and pycover 0.2 [Csillag 1999-07-01]. Neither met all my requirements.

Both have these problems:

• No summary report (requirement 2).

• Annotations aren't accurate (requirement 4):

  • Second and subsequent lines of multi-line statements are marked as not executed.

  • Lines containing only else: are marked as not executed.

  • Comment lines and blank lines are incorrectly recognized and marked as blank if they appear in multi-line strings.

trace.py also has these problems:

• Complicated programmatic interface (requirement 5).

• Substantially slower than other coverage testing tools (requirement 6): see table 1.

A. References

[Csillag 1999-07-01]	"pycover 0.2"; Andrew Csillag; 1999-07-01; <http://www.geocities.com/drew_csillag/pycover.html>.
[Dalke 1999]	"trace.py"; Andrew Dalke; 1999; <ftp://starship.python.net/pub/crew/dalke/trace.py>.
[GDR 2001-12-04]	"Statement coverage for Python"; Gareth Rees; Ravenbrook Limited; 2001-12-04; <http://www.garethrees.org/2001/12/04/python-coverage/>
[van Rossum 2001-07-20]	"Python Reference Manual (release 2.1.1)"; Guido van Rossum; 2001-07-20; <http://www.python.org/doc/2.1.1/lib/lib.html>.

B. Document History

2001-12-04

GDR

Created.

---

Copyright © 2001 Gareth Rees. This document is provided "as is", without any express or implied warranty. In no event will the authors be held liable for any damages arising from the use of this document. You may make and distribute verbatim copies of this document provided that you do not charge a fee for this document or for its distribution.