Ned Batchelder's blog
https://nedbatchelder.com/blog
Ned Batchelder's personal blog.en-USNed Batchelder's blog
https://nedbatchelder.com/blog
https://nedbatchelder.com/pix/rss-banner.gifCoverage 5.0 beta 1
https://nedbatchelder.com//blog/201911/coverage_50_beta_1.html
2019-11-11T18:27:54-05:00Ned BatchelderI want to finish coverage.py 5.0. It has some big changes, so I need people
to try it and tell me if it’s ready. Please install
coverage.py 5.0 beta 1
and try it in your environment.
I especially want to hear from you if you tried the earlier alphas of 5.0.
There have been some changes in the SQLite database that were needed to make
measurement efficient enough for large test suites, but that hinder ad-hoc
querying.
If you haven’t taken a look at coverage.py 5.0 yet, the big change is the
addition of “contexts.” These can record not just that a line was executed, but
something about why it was executed. Any number of contexts can be recorded for
a line. They could be different operating systems, or versions of Python, or
the name of the test that was running. I think it could enable some really
interesting tooling.
If you are interested in recording test names as contexts, the
pytest-cov pytest plugin now
has a “--cov-context” option to do just that.
Contexts increase the data requirements, so data storage is now a SQLite file
rather than a JSON file. The summary of what’s new in 5.0 is here:
Major changes in 5.0.
Please try this. Soon 5.0 will be done, and people will begin installing it
unknowingly. I would really like to minimize the turmoil when that happens.
]]>Debugging TensorFlow coverage
https://nedbatchelder.com//blog/201910/debugging_tensorflow_coverage.html
2019-10-27T17:14:00-04:00Ned BatchelderIt started with a coverage.py issue:
Coverage not working for TensorFlow Model call function.
A line in the code is executing, but coverage.py marks it as unexecuted. How could that be?
TensorFlow was completely new to me. I knew it had some unusual execution
semantics, but I didn’t know what it involved. What could it be doing that
would interfere with coverage measurement? Once I had instructions to reproduce
the issue, I could see that it was true: a line that clearly printed output
during the test run was marked as unexecuted.
The code in question was in a file called routenet_model.py. It had a line
like this:
print('******** in call ******************')
It was the only such line in the code, and sure enough, the test output
showed that “**** in call ****” text, so the line was definitely running.
The first step was to see who was calling the product code. It seemed like
something about a caller was getting in the way, since other code in that file
was marked as executed. I added this to get a stack trace at that point:
importinspect print("\n".join("%30s : %s:%d"%(t[3],t[1],t[2])fortininspect.stack()[::-1])) print('******** in call ******************')
When I re-ran the test, I saw a long stack trace that ended like this (I’ve
abbreviated some of the file paths):
This stack shows the function name, the file path, and the line number in a
compact way. It’s a useful enough debugging helper that I have it as a
vim abbreviation.
Hmm, interesting: there’s a temporary Python file (tmps9vwjn47.py) in the
call stack. That’s definitely unusual. The file is gone by the time the tests
are done, so to get the contents, I grab the filename from the stack trace, and
copy the contents elsewhere:
I named the copied file “weird.py” because a temporary Python file is weird
any time, but this is where it gets really weird: weird.py is a 528-line Python
file, but it doesn’t have the function indicated in the stack trace: there’s
nothing named tf__call in it. The stack trace also indicates that
line 10 is running, but line 10 is a comment:
1 # Copyright 2018 The TensorFlow Authors. All Rights Reserved. 2 # 3 # Licensed under the Apache License, Version 2.0 (the "License"); 4 # you may not use this file except in compliance with the License. 5 # You may obtain a copy of the License at 6 # 7 # http://www.apache.org/licenses/LICENSE-2.0 8 # 9 # Unless required by applicable law or agreed to in writing, software 10 # distributed under the License is distributed on an "AS IS" BASIS, 11 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 # See the License for the specific language governing permissions and 13 # limitations under the License. 14 # ============================================================================== 15 """Control flow statements: loops, conditionals, etc.""" 16 17 from__future__importabsolute_import 18 from__future__importdivision 19 from__future__importprint_function 20 21 ...etc...
Something is truly weird here. To add to the confusion, I can print the
entire FrameInfo from the stack trace, with:
The code_context attribute there shows a plausible line of code, but it
doesn’t correspond to the code in the file at all. This is a twist on a
long-standing gotcha with Python stack traces. When Python code is running, it
has filenames and line numbers in the frames on the call stack, but it doesn’t
keep the source of the code it runs. To populate a stack trace with lines of
code, it reads the file on disk. The classic problem with this is that the file
on disk may have changed since the code started running. So the lines of source
in the stack trace might be wrong because they are newer than the actual code
that is running.
But that’s not what we’re seeing here. Now the line of code in the stack
trace doesn’t match the file on disk at all. It seems to correspond to what is
running, but not what is on disk. The reason is that Python uses a module
called linecache to read the line of source. As the name implies, linecache
caches file contents, so that reading many different lines from a file won’t try
to open the same file many times.
What must have happened here is that the file had one program in it, and then
was read (and cached) by linecache for some reason. That first program is what
is running. Then the file was re-written with a second program. Linecache
checks the modification time to invalidate the cache, but if the file was
rewritten quickly enough to not have a different modification time, then the
stale cache would be used. This is why the stack trace has the correct line of
code, even though the file on disk doesn’t.
A quick look in the __pycache__ directory in the tmp directory shows a .pyc
file, and if I dump it with show_pyc.py,
I can see that it has the code I’m interested in. But rather than try to read
disassembled bytecode, I can get the source from the stale copy in
linecache!
This is the code from our original routenet_model.py (including all the
debugging code that I put in there), translated into some kind of annotated
form. The reason coverage.py said the product code wasn’t run is because it
wasn’t run! A copy of the code was run.
Now I realize something about inspect.stack(): the first frame it shows is
your caller. If I had used a stack trace that showed the current frame first,
it would have shown that my debugging code was not in the file I thought it
was.
It turns out that inspect.stack() is a one-line helper using other things:
defstack(context=1): """Return a list of records for the stack above the caller's frame.""" returngetouterframes(sys._getframe(1),context)
Changing my stack trace one-liner to use getoutframes(sys._getframe()) is
better, but is still confusing in this case because TensorFlow rewrites function
calls, including sys._getframe, so the resulting stack trace ends with:
Even now, I can’t quite wrap my head around why it comes out that way.
The next step is to decide what to do about this. The converted code has a
parameter called ag_source_map__, which is a map from converted code back to
source code. This could be used to get the coverage right, perhaps in a plugin,
but I need to hear from TensorFlow people to see what would be the best
approach. I’ve written a
TensorFlow
issue to start the conversation.
]]>Pytest-cov support for who-tests-what
https://nedbatchelder.com//blog/201910/pytestcov_support_for_whotestswhat.html
2019-10-08T19:14:09-04:00Ned BatchelderI’ve added a new option to the pytest-cov
coverage plugin for pytest: --cov-context=test will set the dynamic context
based on pytest test phases. Each test has a setup, run, and teardown phase.
This gives you the best test information in the coverage database:
The full test id is used in the context. You have the test file name, and
the test class name if you are using class-based tests.
Parameterized tests start a new context for each new set of parameter
values.
Execution is a little faster because coverage.py doesn’t have to poll for
test starts.
For example, here is a repo of simple pytest tests in a number of forms:
pytest-gallery. I can
run the tests with test contexts being recorded:
Version 2.8.0 of pytest-cov (and later) has the new feature. Give it a try.
BTW, I also snuck another alpha of coverage.py (5.0a8) in at the same time, to
get a needed API right.
Still missing from all this is a really useful way to report on the data.
Get in touch if you have needs or ideas.
]]>Sponsor me on GitHub?
https://nedbatchelder.com//blog/201910/sponsor_me_on_github.html
2019-10-07T08:52:13-04:00Ned Batcheldertl;dr: You can
sponsor me on GitHub,
but I’m not sure why you would.
In May, GitHub launched
GitHub Sponsors,
a feature on their site for people to support each other financially. It’s
still in beta, but now I’m in the program, so
you can sponsor me
if you want.
I’m very interested in the question of how the creators of open source
software can benefit more from what they create, considering how much value
others get from it.
To be honest, I’m not sure GitHub Sponsors is going to make a big
difference. It’s another form of what I’ve called an
internet tip jar: it focuses on one
person giving another person money. Don’t get me wrong: I’m all for enabling
interpersonal connections of all sorts. But I don’t think that will scale to
improve the situation meaningfully.
I’m participating in GitHub Sponsors because I want to try every possible
avenue. Since it’s on GitHub, it will get more attention than most tip jars,
so maybe it will work out differently. Participating is a good way for me to
understand it.
GitHub lets me define tiers of sponsorship, with different incentives,
similar to Kickstarter. I don’t know what will motivate people, and I don’t
have existing incentives at my fingertips to offer, so I’ve just created
three generic tiers
($3, $10, $30 per month). If GitHub Sponsors appeals to you, let me know what
I could do with a tier that might attract other people.
The question mark in the title is not because I’m making a request of you.
It’s because I’m uncertain whether and why people will become sponsors through
GitHub Sponsors. We’ll see what happens.
]]>Coverage.py 5.0a7, and the future of pytest-cov
https://nedbatchelder.com//blog/201909/coveragepy_50a7_and_the_future_of_pytestcov.html
2019-09-24T09:58:46-04:00Ned BatchelderProgress continues in the Python coverage world. Two recent things: first,
the latest alpha of Coverage.py 5.0 is available:
5.0a7. Second,
pytest-cov is supporing coverage.py 5.0, and we’re talking about the future of
pytest-cov.
There are two big changes in Coverage.py 5.0a7. First, there is a new
reporting command: coverage json produces a JSON file with
information similar to the XML report. In coverage.py 4.x, the data storage was
a lightly cloaked JSON file. That file was not in a supported format, and in
fact, it is gone in 5.0. This command produces a supported JSON format for
people who want programmatic access to details of the coverage data. A huge
thanks to Matt Bachmann for implementing it.
The second big change is to the SQL schema in the 5.x data file, which is a
SQLite database. Previously, each line measured produced a row in the “line”
table. But this proved too bulky for large projects. Now line numbers are
stored in a compact binary form. There is just one row in the “line_bits” table
for each file and context measured. This makes it more difficult to use the
data with ad-hoc queries. Coverage provides
functions for working with the line number bitmaps,
but I’m interested in other ideas about how to make the data more usable.
The pytest-cov changes are to support coverage.py 5.0. Those changes are
already on the master branch.
I’m also working on a pull request to add a
--cov-contexts=test option
so that pytest can announce when tests change, for accurate and detailed dynamic
contexts.
Longer-term, I’d like to shrink the size of the pytest-cov plugin. Pytest
should be about running tests, not reporting on coverage after the tests are
run. Too much of the code, and too many of the bug reports, are due to it
trying to take on more than it needs to. The command-line arguments are getting
convoluted, for no good reason. I’ve written an issue to get feedback:
Proposal:
pytest-cov should do less. If you have opinions one way or the other, that
would be a good place to talk about them.
]]>Variable fonts
https://nedbatchelder.com//blog/201909/variable_fonts.html
2019-09-17T19:30:45-04:00Ned BatchelderWe’re all used to fonts coming in different weights (normal, bold), or
sometimes different widths (normal, condensed, extended). Geometrically,
there’s no reason that these variations need to be discrete. It’s a limitation
of technology that we’ve been given a few specific weights or widths to choose
from.
Over the years there have been a few attempts to make those variation
dimensions continuous rather than discrete.
Knuth’s Metafont was one,
Adobe’s Multiple Master fonts
were another. The latest is OpenType’s variable fonts.
In a variable font, the type designer not only decides on the shapes of the
glyphs, but on the axes of variability. Weight and width are two obvious ones,
but the choice is arbitrary.
One of the great things about variable fonts is that browsers have good
support for them. You can use a variable font on a web page, and set the values
of the variability dimensions using CSS.
Browser support also means you can play with the variability without any special
tools. Nick Sherman’s v-fonts.com is a gallery
and playground of variable fonts. Each is displayed with sliders for its axes.
You can drag the sliders and see the font change in real time in your browser.
Many of the fonts are gimmicky, either to show off the technology, or because
exotic display faces are where variability can be used most broadly. But here
are a few of my choices that demonstrate variability to its best advantage:
Antonia Variable
includes an optical size axis. Optical size refers to the adjustments that have
to be made to shapes to compensate for the size of the font. At tiny sizes,
letters have to be wider, and features sturdier in order for type to remain
legible, but also seem like the same family. It’s kind of like how babies have
the same features as adults, but smaller and plumper.
UT Morph is an
ultra-geometric display face with two stark axes, positive and negative. This
shows how variability can be used to control completely new aspects of a
design.
Recursive has some really
interesting axes that use variability in eye-opening ways without being
cartoonish: proportion (how monospaced is it), expression (how swoopy is it),
and italic (changes a few letter shapes).
Variable fonts are still a new technology, but we’ll see them being used more
and more. Don’t expect to see fonts stretching and squashing before your eyes
though. Site designers will use variability to make some choices, and you won’t
even realize variability was involved. Like all good typography, it won’t draw
attention to itself.
]]>Don’t omit tests from coverage
https://nedbatchelder.com//blog/201908/dont_omit_tests_from_coverage.html
2019-08-29T06:43:59-04:00Ned BatchelderThere’s a common idea out there that I want to refute. It’s this: when
measuring coverage, you should omit your tests from measurement.
Searching GitHub shows that lots of people do this.
This is a bad idea. Your tests are real code, and the whole
point of coverage is to give you information about your code. Why wouldn’t you
want that information about your tests?
You might say, “but all my tests run all their code, so it’s useless
information.” Consider this scenario: you have three tests written, and you need
a fourth, similar to the third. You copy/paste the third test, tweak the
details, and now you have four tests. Except oops, you forgot to change the
name of the test.
Tests are weird: you have to name them, but the names don’t matter. Nothing
calls the name directly. It’s really easy to end up with two same-named tests.
Which means you only have one test, because the new one overwrites the old.
Coverage would alert you to the problem.
Also, if your test suite is large, you likely have helper code in there as
well as straight-up tests. Are you sure you need all that helper code? If you
run coverage on the tests (and the helpers), you’d know about some weird clause
in there that is never used. That’s odd, why is that? It’s probably useful to
know. Maybe it’s a case you no longer need to consider. Maybe your tests aren’t
exercising everything you thought.
The only argument against running coverage on tests is that it “artificially”
inflates the results. True, it’s much easier to get 100% coverage on a test
file than a product file. But so what? Your coverage goal was chosen arbitrarily
anyway. Instead of aiming for 90% coverage, you should include your tests and
aim for 95% coverage. 90% doesn’t have a magical meaning.
What’s the downside of including tests in coverage? “People will write more
tests as a way to get the easy coverage.” Sounds good to me. If your developers
are trying to game the stats, they’ll find a way, and you have bigger
problems.
True, it makes the reports larger, but if your tests are 100% covered,
you can exclude those files from the report with
[report] skip_covered
setting.
Your tests are important. You’ve put significant work into them. You want to
know everything you can about them. Coverage can help. Don’t omit tests from
coverage.
]]>Why your mock doesn’t work
https://nedbatchelder.com//blog/201908/why_your_mock_doesnt_work.html
2019-08-02T14:15:00-04:00Ned BatchelderMocking is
a powerful technique for isolating tests from undesired interactions among
components. But often people find their mock isn’t taking effect, and it’s not
clear why. Hopefully this explanation will clear things up.
BTW: it’s really easy to over-use mocking. These are good explanations of
alternative approaches:
Before we get to fancy stuff like mocks, I want to review a little bit about
Python assignment. You may already know this, but bear with me. Everything
that follows is going to be directly related to this simple example.
Variables in Python are names that refer to values. If we assign a second
name, the names don’t refer to each other, they both refer to the same value.
If one of the names is then assigned again, the other name isn’t affected:
If this is unfamiliar to you, or you just want to look at more pictures like
this, Python Names and Values goes into much more depth about the semantics
of Python assignment.
Importing
Let’s say we have a simple module like this:
# mod.py
val="original"
defupdate_val(): globalval val="updated"
We want to use val from this module, and also call update_val to change val.
There are two ways we could try to do it. At first glance, it seems like they
would do the same thing.
The first version imports the names we want, and uses them:
# code1.py
frommodimportval,update_val
print(val) update_val() print(val)
The second version imports the module, and uses the names as attributes on
the module object:
# code2.py
importmod
print(mod.val) mod.update_val() print(mod.val)
This seems like a subtle distinction, almost a stylistic choice. But
code1.py prints “original original”: the value hasn’t changed! Code2.py does
what we expected: it prints “original updated.” Why the difference?
Let’s look at code1.py more closely:
# code1.py
frommodimportval,update_val
print(val) update_val() print(val)
After “from mod import val”, when we first print val, we have this:
“from mod import val” means, import mod, and then do the assignment “val =
mod.val”. This makes our name val refer to the same object as mod’s name
val.
After “update_val()”, when we print val again, our world looks like this:
update_val has reassigned mod’s val, but that has no effect on our val. This
is the same behavior as our x and y example, but with imports instead of more
obvious assignments. In code1.py, “from mod import val” is an assignment from
mod.val to val, and works exactly like “y = x” does. Later assignments to
mod.val don’t affect our val, just as later assignments to x don’t affect y.
Now let’s look at code2.py again:
# code2.py
importmod
print(mod.val) mod.update_val() print(mod.val)
The “import mod” statement means, make my name mod refer to the entire mod
module. Accessing mod.val will reach into the mod module, find its val name, and
use its value.
Then after “update_val()”, mod’s name val has been changed:
Now we print mod.val again, and see its updated value, just as we expected.
OK, but what about mocks?
Mocking is a fancy kind of assignment: replace an object (or function) with
a different one. We’ll use the
mock.patch
function in a with statement. It makes a mock object, assigns it to the name
given, and then restores the original value at the end of the with
statement.
Let’s consider this (very roughly sketched) product code and test:
# product.py
fromosimportlistdir
defmy_function(): files=listdir(some_directory) # ... use the file names ...
After we’ve imported product.py, both the os module and product.py have a
name “listdir” which refers to the built-in listdir() function. The references
look like this:
The mock.patch in our test is really just a fancy assignment to the name
“os.listdir”. During the test, the references look like this:
You can see why the mock doesn’t work: we’re mocking something, but it’s not
the thing our product code is going to call. This situation is exactly analogous
to our code1.py example from earlier.
You might be thinking, “ok, so let’s do that code2.py thing to make it work!”
If we do, it will work. Your product code and test will now look like this (the
test code is unchanged):
# product.py
importos
defmy_function(): files=os.listdir(some_directory) # ... use the file names ...
When the test is run, the references look like this:
Because the product code refers to the os module, changing the name in the
module is enough to affect the product code.
But there’s still a problem: this will mock that function for any module
using it. This might be a more widespread effect than you intended. Perhaps
your product code also calls some helpers, which also need to list files. The
helpers might end up using your mock (depending how they imported os.listdir!),
which isn’t what you wanted.
Mock it where it’s used
The best approach to mocking is to mock the object where it is used, not
where it is defined. Your product and test code will look like this:
# product.py
fromosimportlistdir
defmy_function(): files=listdir(some_directory) # ... use the file names ...
The only difference here from our first try is that we mock “product.listdir”,
not “os.listdir”. That seems odd, because listdir isn’t defined in
product.py. That’s fine, the name “listdir” is in both the os module and in
product.py, and they are both references to the thing you want to mock. Neither
is a more real name than the other.
By mocking where the object is used, we have tighter control over what
callers are affected. Since we only want product.py’s behavior to change, we
mock the name in product.py. This also makes the test more clearly tied to
product.py.
As before, our references look like this once product.py has been fully
imported:
The difference now is how the mock changes things. During the test, our references look like this:
The code in product.py will use the mock, and no other code will. Just what
we wanted!
Is this OK?
At this point, you might be concerned: it seems like mocking is kind of
delicate. Notice that even with our last example, how we create the mock
depends on something as arbitrary as how we imported the function. If our code
had “import os” at the top, we wouldn’t have been able to create our mock
properly. This is something that could be changed in a refactoring, but at
least mock.patch will fail in that case.
You are right to be concerned: mocking is delicate. It depends on
implementation details of the product code to construct the test. There are
many reasons to be wary of mocks, and there are other approaches to solving the
problems of isolating your product code from problematic dependencies.
If you do use mocks, at least now you know how to make them work, but again,
there are other approaches. See the links at the top of this page.
]]>Set_env.py
https://nedbatchelder.com//blog/201907/set_envpy.html
2019-07-21T13:36:21-04:00Ned BatchelderA good practice when writing complicated software is to put in lots of
debugging code. This might be extra logging, or special modes that tweak the
behavior to be more understandable, or switches to turn off some aspect of your
test suite so you can focus on the part you care about at the moment.
But how do you control that debugging code? Where are the on/off switches?
You don’t want to clutter your real UI with controls. A convenient option is
environment variables: you can access them simply in the code, your shell has
ways to turn them on and off at a variety of scopes, and they are invisible to
your users.
Though if they are invisible to your users, they are also invisible to you!
How do you remember what exotic options you’ve coded into your program, and how
do you easily see what is set, and change what is set?
I’ve been using environment variables like this in coverage.py for years, but
only recently made it easier to work with them.
To do that, I wrote set_env.py.
It scans a tree of files for special comments describing environment variables,
then shows you the values of those variables. You can type quick commands to
change the values, and when the program is done, it updates your environment.
It’s not a masterpiece of engineering, but it works for me.
As an example, this line appears in coverage.py:
# $set_env.py: COVERAGE_NO_PYTRACER - Don't run the tests under the Python tracer.
This line is found by set_env.py, so it knows that COVERAGE_NO_PYTRACER is
one of the environment variables it should fiddle with.
When I run set_env.py in the coverage.py tree, I get something like this:
$ set_env Read 298 files 1: COVERAGE_AST_DUMP Dump the AST nodes when parsing code. 2: COVERAGE_CONTEXT Set to 'test_function' for who-tests-what 3: COVERAGE_DEBUG Options for --debug. 4: COVERAGE_DEBUG_CALLS Lots and lots of output about calls to Coverage. 5: COVERAGE_ENV_ID Use environment-specific test directories. 6: COVERAGE_KEEP_TMP Keep the temp directories made by tests. 7: COVERAGE_NO_CONTRACTS Disable PyContracts to simplify stack traces. 8: COVERAGE_NO_CTRACER Don't run the tests under the C tracer. 9: COVERAGE_NO_PYTRACER = '1' Don't run the tests under the Python tracer. 10: COVERAGE_PROFILE Set to use ox_profile. 11: COVERAGE_TRACK_ARCS Trace every arc added while parsing code. 12: PYTEST_ADDOPTS Extra arguments to pytest.
(# [value] | x # ... | ? | q)>
All of the files were scanned, and 12 environment variables found. We can
see that COVERAGE_NO_PYTRACER has the value “1”, and none of the others are in
the environment. At the prompt, if I type “4”, then COVERAGE_DEBUG_CALLS (line
4) will be toggled to “1”. Type “4” again, and it is cleared. Typing “4 yes
please” will set it to “yes please”, but often I just need something or
nothing, so toggling “1” as the value works.
One bit of complexity here is that a program you run in your shell can’t
change environment variables for subsequent programs, which is exactly what we
need. So “set_env” is actually a shell alias:
aliasset_env='$(set_env.py $(git ls-files))'
This runs set_env.py against all of the files checked-in to git, and then
executes whatever set_env.py outputs. Naturally, set_env.py outputs shell
commands to set environment variables. If ls-files produces too much output,
you can use globs there also, so “**/*.py” might be useful.
Like I said, it’s not a masterpiece, but it works for me. If there are other
tools out there that do similar things, I’d like to hear about them.
]]>Coverage.py 5.0a6: context reporting
https://nedbatchelder.com//blog/201907/coveragepy_50a6_context_reporting.html
2019-07-17T07:25:00-04:00Ned BatchelderI’ve released another alpha of coverage.py 5.0:
coverage.py 5.0a6.
There are some design decisions ahead that I could use feedback on.
Important backstory:
The big feature in 5.0 is “contexts”: recording data for varying execution
context, also known as Who Tests
What. The idea is to record not just that a line was executed, but also
which tests ran each line.
Some of the changes in alpha 6 were driven by a hackathon project at work:
using who-tests-what on the large Open edX codebase. We wanted to collect
context information, and then for each new pull request, run only the subset of
tests that touched the lines you changed. Initial experiments indicate this
could be a huge time-savings.
Big changes in this alpha:
Support for contexts when reporting. The --show-contexts option annotates
lines with the names of contexts recorded for the line. The --contexts option
lets you filter the report to only certain contexts. Big thanks to Stephan
Richter and Albertas Agejevas for the contribution.
Our largest test suite at work has 29k tests. The .coverage SQLite data
file was 659Mb, which was too large to work with. I changed the database format
to use a compact bitmap representation for line numbers, which reduced the data
file to 69Mb, a huge win.
The API to the CoverageData object has changed.
Some implications of these changes:
The HTML reporting on contexts is good for small test suites, but very
quickly becomes unwieldy if you have more than 100 tests. Please try using it
and let me know what kind of reporting would be helpful.
The new more-compact data file is harder to query. The larger data file has
a schema designed to be useful for ad-hoc querying. It was a classic
third-normal form representation of the data. Now I consider the database
schema to be a private implementation detail. Should we have a new “coverage
sql” report command that exports the data to a convenient SQLite file?
Because CoverageData has changed, you will need an updated version of
pytest-cov if you use that plugin. The future of the plugin is somewhat up in
the air. If you would like to help maintain it, get in touch. You can install
the up-to-date code with:
To support our hackathon project, we wrote a new pytest plugin: it uses
pytest hooks to indicate the test boundaries, and can read the database and the
code diff to choose the subset of tests to run. This plugin is in very
rough shape (as in, it hasn’t yet fully worked), but if you are interested
in participating in this experiment, get in touch. The code is here
nedbat/coverage_pytest_plugin.
I don’t think this will remain as an independent plugin, so again, if you want
to help with future maintenance or direction, let me know.
All of our experimentation (and improvements) for contexts involve line
coverage. Branch coverage only complicates the problems of storage and
reporting. I’ve mused
about how to store branch data more compactly in the past, but nothing has
been done.
I know this is a lot, and the 5.0 alpha series has been going on for a while.
The features are shaping up to be powerful and useful. All of your feedback has
been very helpful, keep it coming.
