Sunday 7 October 2018 — This is six years old. Be careful.
A long-awaited feature of coverage.py is now available in a rough form: Who Tests What annotates coverage data with the name of the test function that ran the code.
To try it out:
- Install coverage.py v5.0a3.
- Add this line literally to the [run] section of your .coveragerc file:
- Run your tests.
- The .coverage file is now a SQLite database. There is no change to reporting yet, so you will need to do your own querying of the SQLite database to get information out. See below for a description of the database schema.
[run]
dynamic_context = test_function
The database can be accessed in any SQLite-compatible way you like. Note that the schema is not (yet) part of the public API. That is, it may not be guaranteed to stay the same. This is one of the things yet to be decided. For now though, the database has these tables:
- file: maps full file paths to file_ids: id, path
- context: maps contexts (test function names) to contexts_ids: id, context
- line: the line execution data: file_id, context_id, lineno
- arc: similar to line, but for branch coverage: file_id, context_id, fromno, tono
It’s not the most convenient, but the information is all there. If you used branch coverage, then the important data is in the “arc” table, and “line” is empty. If you didn’t use branch coverage, then “line” has data and “arc” is empty. For example, using the sqlite3 command-line tool, here’s a query to see which tests ran a particular line:
sqlite> select
...> distinct context.context from arc, file, context
...> where arc.file_id = file.id
...> and arc.context_id = context.id
...> and file.path like '%/xmlreport.py'
...> and arc.tono = 122;
context
------------------------------------------------------------
XmlPackageStructureTest.test_package_names
OmitIncludeTestsMixin.test_omit
OmitIncludeTestsMixin.test_include
OmitIncludeTestsMixin.test_omit_2
XmlReportTest.test_filename_format_showing_everything
XmlReportTest.test_no_source
OmitIncludeTestsMixin.test_include_as_string
OmitIncludeTestsMixin.test_omit_and_include
XmlReportTest.test_empty_file_is_100_not_0
OmitIncludeTestsMixin.test_omit_as_string
XmlReportTest.test_nonascii_directory
OmitIncludeTestsMixin.test_nothing_specified
XmlReportTest.test_curdir_source
XmlReportTest.test_deep_source
XmlPackageStructureTest.test_package_depth
XmlPackageStructureTest.test_source_prefix
XmlGoldTest.test_a_xml_2
XmlGoldTest.test_a_xml_1
XmlReportTest.test_filename_format_including_module
XmlReportTest.test_reporting_on_nothing
XmlReportTest.test_filename_format_including_filename
ReportingReturnValueTest.test_xml
OmitIncludeTestsMixin.test_include_2
BTW, there are also “static contexts” if you are interested in keeping coverage data from different test runs separate: see Measurement Contexts in the docs for details.
Some things to note and think about:
- The test function name recorded includes the test class if we can figure it out. Sometimes this isn’t possible. Would it be better to record the filename and line number?
- Is test_function too fine-grained for some people? Maybe chunking to the test class or even the test file would be enough?
- Better would be to have test runner plugins that could tell us the test identifier. Anyone want to help with that?
- What other kinds of dynamic contexts might be useful?
- What would be good ways to report on this data? How are you navigating the data to get useful information from it?
- How is the performance?
- We could have a “coverage extract” command that would be like the opposite of “coverage combine”: it could pull out a subset of the data so a readable report could be made from it.
Please try this out, and let me know how it goes. Thanks.
Comments
I wanted to mention that "test_function" seems like an appropriately fine-grained depth. In my line of work I deal with hardware platforms that have minor behavioral differences between firmware versions. Consequently, there are many places where a conditional is only true for a specific firmware version, so when I write unit tests, each one typically triggers one specific corner case.
Being able to confirm which individual test function is triggering a specific path seems really helpful. In comparison, chunking to the test class or test file would not be helpful in my specific use case.
I don't have other feedback beyond describing a little bit of my use case. Thank you again for all of your work on coverage!
Great to hear that "Who tests what?" is ready! Thanks a lot for all the effort put into this essential tool!
I'm pretty sure it will ease implementation of python-tia, a generic Test Impact Analysis (TIA) preprocessor for test tools and static analyzers.
> The test function name recorded includes the test class if we can figure it out. Sometimes this isn’t possible. Would it be better to record the filename and line number?
In case this is an either/or decision I'd prefer test_class.test_function. Would it be too costly to add filename and line number?
> Is test_function too fine-grained for some people? Maybe chunking to the test class or even the test file would be enough?
No. I need the test_function. Having the information about the full path (test_file.test_class.test_function) could be great for further processing.
> Better would be to have test runnner plugins that could tell us the test identifier. Anyone want to help with that?
I'd prefer coverage.py beeing independent on test runners. However this is not possible if we want to get other dynamic contexts. Helping out... depends on spare time and relevance for python-tia.
> What other kinds of dynamic contexts might be useful?
W.r.t. improvement of the granularity of test impact analysis I thought about how to get information about the mapping of tests to their run-time dependencies like pytest fixtures a test depends on, test utility functions a test depends on, etc. .
> What would be good ways to report on this data? How are you navigating the data to get useful information from it?
Reporting is not of primary interest for me. Accessing the stored data from a SQLite file (data location and internal representation) is just fine for me. With python-tia I'd like to stick to a functional programming approach with corresponding data structures. Means I'd read the SQLite data, output an iterable sequence of e.g. namedtuples and work with this.
> How is the performance?
Let's see :)
> We could have a “coverage extract” command that would be like the opposite of “coverage combine”: it could pull out a subset of the data so a readable report could be made from it.
Having a command line command is a good idea. For python-tia I'd prefer to have an API to access the coverage.py SQLite data.
I had some tests that were your standard unit tests gathering some coverage data with the parallel option set. Then I had functional tests that excercised the app running in a Docker container with enabled coverage measurement. After the tests I would copy the partial coverage information down from the container. The problem was that the code paths in the local tests were different from those in the container, so I had to replace them in .coverage file from the container before running "coverage combine".
Is there a nice way to work around this issue that's built into coverage that I don't know about? If not, then should I just be doing the same trick, but with altering / merging SQLite databases?
But coverage is run using the https://coverage.readthedocs.io/en/v4.5.x/subprocess.html#subprocess COVERAGE_PROCESS_START env var, so I cannot set the static context via cli. Would it be possible for you to add option for setting the static context via an environment variable and/or an argument to coverage.process_startup().
I realize this is not a place to file feature request so I can create a github issue if you say.
A comannd line option on the coverage html would be good to turn off this drill down (off being the default).
Now that I've played with the alpha a little bit, I'm trying to figure out what features I want in terms of dynamic contexts. For reference, this is an attempt to replicate the output of a rather slow plugin/extension to Coverage that I wrote.
The current behavior of the plugin that drives this is to send messages to a modified coverage runner that cause it to kick off new runs at various points. I could imagine asking for something that approximates this interface, but it doesn't seem like something that fits well with the design.
I then imagined asking for the ability to write custom functions to generate contexts, but that seems like it would slow down tracing considerably.
Because for my purposes, the context required by a function can be worked out ahead of time, I think one possibility that would work for me would be the ability to dynamically "register" a function as having a particular context. Then choosing a context would basically be a matter of checking a mapping for the context name (a json-encoded data structure, in this case), and entering it if it exists.
Add a comment: