« | » Main « | »

Using context managers in test setUp

Monday 17 August 2015

Python's context managers are the general mechnism underlying the "with" statement. They're a nice abstraction of doing a thing, and then later undoing it. The classic example is opening a file and then later closing it:

with open("some_file.txt") as f:
    # do things with f..

# f is closed by the time you get here.

But context managers can be used for other do-then-later-undo types of behavior. Here's a context manager that changes the current directory, then later changes it back:

import contextlib

@contextlib.contextmanager
def change_dir(new_dir):
    """Change directory, and then change back.

    Use as a context manager, it will give you the new directory, 
    and later restore the old one.

    """
    old_dir = os.getcwd()
    os.chdir(new_dir)
    try:
        yield os.getcwd()
    finally:
        os.chdir(old_dir)

...

with change_dir("/path/to/new/dir"):
    # do something while in the new directory.

Context managers are objects that have __enter__ and __exit__ methods, but here we've used a very handy decorator from contextlib to make a context manager using the yield statement.

Now, suppose you have a context manager that neatly encapsulates your needed behavior, and further suppose that you are writing unit tests, and wish to get this behavior in your setUp and tearDown methods. How do you do it?

You can't use a with statement, because you need the "do" part of the context manager to happen in setUp, and then you need the "later undo" part of it to happen in tearDown. The syntax-indicated scope of the with statement won't work.

You can do it using the context manager protocol directly to perform the actions you need. And unittest has a mechanism better than tearDown: addCleanup takes a callable, and guarantees to call it when the test is done. So both the before-test logic and the after-test logic can be expressed in one place.

Here's how: write a helper function to use a context manager in a setUp function:

def setup_with_context_manager(testcase, cm):
    """Use a contextmanager to setUp a test case."""
    val = cm.__enter__()
    testcase.addCleanup(cm.__exit__, None, None, None)
    return val

Now where you would have used a context manager like this:

with ctxmgr(a, b, c) as v:
    # do something with v

you can do this in your setUp function:

def setUp(self):
    self.v = setup_with_context_manager(self, ctxmgr(a, b, c))

def test_foo(self):
    # do something with self.v

Simple and clean.

Notice that @contextlib.contextmanager lets us write a generator, then use a decorator to turn it into a context manager. There's a lot of Python features at work here in a very small space, which is kind of cool. Then we use addCleanup to take a callable as a first-class object to get the clean up we need, which is even more cool.

One caveat about this technique: a context manager's __exit__ method can be called with information about an exception in progress. The mechanism shown here will never do that. I'm not sure it even should, considering how it's being used in a test suite. But just beware.

Small choices, big decisions: coverage run --append

Wednesday 5 August 2015

A seemingly simple change to fix a small bug lead me to some interesting software design choices. I'll try to explain.

In the new beta of coverage.py, I had a regression where the "run --append" option didn't work when there wasn't an existing data file. The problem was code in class CoverageScript in cmdline.py that looked like this:

if options.append:
    self.coverage.combine(".coverage")
self.coverage.save()

If there was no .coverage data file, then this code would fail. The fix was really simple: just check if the file exists before trying to combine it:

if options.append:
    if os.path.exists(".coverage"):
        self.coverage.combine(".coverage")
self.coverage.save()

(Of course, all of these code examples have been simplified from the actual code...)

The problem with this has to do with how the CoverageScript class is tested. It's responsible for dealing with the command-line syntax, and invoking methods on a coverage.Coverage object. To make the testing faster and more focused, test_cmdline.py uses mocking. It doesn't use an actual Coverage object, it uses a mock, and checks that the right methods are being invoked on it.

The test for this bit of code looked like this, using a mocking helper that works from a sketch of methods being invoked:

self.cmd_executes("run --append foo.py", """\
    .Coverage()
    .start()
    .run_python_file('foo.py', ['foo.py'])
    .stop()
    .combine('.coverage')
    .save()
    """, path_exists=True)

This test means that "run --append foo.py" will make a Coverage object with no arguments, then call cov.start(), then cov.run_python_file with two arguments, etc.

The problem is that the product code (cmdline.py) will actually call os.path.exists, and maybe call .combine, depending on what it finds. This mocking test can't easily take that into account. The design of cmdline.py was that it was a thin-ish wrapper over the methods on a Coverage object. This made the mocking strategy straightforward. Adding logic in cmdline.py makes the testing more complicated.

OK, second approach: change Coverage.combine() to take a missing_ok=True parameter. Now cmdline.py could tell combine() to not freak out if the file didn't exist, and we could remove the os.path.exists conditional from cmdline.py. The code would look like this:

if options.append:
    self.coverage.combine(".coverage", missing_ok=True)
self.coverage.save()

and the test would now look like this:

self.cmd_executes("run --append foo.py", """\
    .Coverage()
    .start()
    .run_python_file('foo.py', ['foo.py'])
    .stop()
    .combine('.coverage', missing_ok=True)
    .save()
    """, path_exists=True)

Coverage.combine() is part of the public API to coverage.py. Was I really going to extend that supported API for this use case? It would mean documenting, testing, and promising to support that option "forever". There's no nice way to add an unsupported argument to a supported method.

Extending the supported API to simplify my testing seemed like the tail wagging the dog. I'm all for letting testing concerns inform a design. Often the tests are simply proxies for the users of your API, and what makes the testing easier will also make for a better, more modular design.

But this just felt like me being lazy. I didn't want combine() to have a weird option just to save the caller from having to check if the file exists. I imagined explaining this option to someone else, and I didn't want my future self to have to sheepishly admit, "yeah, it made my tests easier..."

What finally turned me back from this choice was the principle of saying "no." Sometimes the best way to keep a product simple and good is to say "no" to extraneous features. Setting aside all the testing concerns, this option on Coverage.combine() just felt extraneous.

Having said "no" to changing the public API, it's back to a conditional in cmdline.py. To make testing CoverageScript easier, I use dependency injection to give the object a function to check for files. CoverageScript already had parameters on the constructor for this purpose, for example to get the stand-in for the Coverage class itself. Now the constructor will look like:

class CoverageScript(object):
    """The command-line interface to coverage.py."""

    def __init__(self, ..., _path_exists=None):
        ...
        self.path_exists = _path_exists or os.path.exists

    def do_run(self, options, args):
        ...

        if options.append:
            if self.path_exists(".coveragerc"):
                self.coverage.combine(".coveragerc")
        self.coverage.save()

and the test code can provide a mock for _path_exists and check its arguments:

self.cmd_executes("run --append foo.py", """\
    .Coverage()
    .start()
    .run_python_file('foo.py', ['foo.py'])
    .stop()
    .path_exists('.coverage')
    .combine('.coverage')
    .save()
    """, path_exists=True)

Yes, this makes the testing more involved. But that's my business, and this doesn't change the public interface in ways I didn't like.

When I started writing this blog post, I was absolutely certain I had made the right choice. As I wrote it, I wavered a bit. Would missing_ok=True be so bad to add to the public interface? Maybe not. It's not such a stretch, and a user of the API might plausibly convince me that it's genuinely helpful to them. If that happens, I can reverse all this. That would be ok too. Decisions, decisions...

Coverage.py 4.0b1

Sunday 2 August 2015

I think Coverage.py v4.0 is ready. But to be on the safe side, I'm releasing it as a beta because there have been a ton of changes since v3.7.1. Try it: coverage.py 4.0b1.

Changes since 4.0a6:

  • The data storage has been completely revamped. There's a new API to access the stored data: coverage.CoverageData.
  • The XML report has a new missing-branches attribute which breaks conformance to the Cobertura DTD.
  • Missing branches in the HTML report now have a bit more information in the right-hand annotations. Hopefully this will make their meaning clearer.
  • The private method Coverage._harvest_data is gone. Some third-party tools relied on this. Please test your integrations.
  • The speed is back to 3.7.1 levels.

If you are interested, there is a complete list of changes: CHANGES.txt.

Also available is the latest version of the Django coverage plugin: django_coverage_plugin 0.6. This uses the new plugin support in Coverage.py 4.0 to implement coverage measurement of Django templates.

« | » Main « | »