« | » Main « | »

Coverage.py 4.0

Sunday 20 September 2015

After more than 20 months, coverage.py 4.0 is available. This version brings a number of significant changes:

  • Plugin support for measuring non-Python files, such as Django templates.
  • Support for concurrency libraries such as gevent, greenlet, and eventlet.
  • Live filtering in the HTML report.
  • More than 50 issues closed, including improved branch coverage.
  • Documentation is now on Read The Docs.

The full changes are in the docs.

A number of things internal to coverage.py have changed in incompatible ways. Helper tools sometimes build on internal interfaces. If you find that coverage.py 4.0 seems broken in some way, please update to the latest versions of your helper libraries while diagnosing the problem.

If you have any problems, please let me know.

Next up for coverage.py: some form of the often requested feature, "show who tests what". If you are interested in that, get in touch, or comment on the issue.


Monday 14 September 2015

I've just done a bunch of work on continuous integration for coverage.py. The biggest change is that now I've got Appveyor running Windows tests and building Windows kits.

Appveyor is a good service: the docs are helpful, and the support forum seems to be an obsession with the staff, especially Feodor Fitsner, who have always answered my questions within 12 hours regardless of when I ask them.

Oliver Grisel has a demonstration of building wheels on Appveyor which was very helpful in getting started.

Of course, part of the problem with supporting Windows is that it is unfamiliar to many of us. Appveyor provides the platform on which to run, but we still have to come up with the steps ourselves. And Python is a bit unfamiliar to Appveyor, so the steps include installing Python. It all gets a bit tangled.

The high point in my adventure came with this line:

 - "python -c \"import os; open('python{0}.{1}.bat'.format(*os.environ['TOXENV'][2:]), 'w').write('@{0}\\\\python \\x25*\\n'.format(os.environ['PYTHON']))\""

Explanation: like most CI services, Appveyor is configured with a YAML file. This line is part of the install step before tests are run. It's a Windows command line. Our appveyor.yml file installs a number of versions of Python, because Appveyor doesn't have all the versions we need pre-installed. So each job sets two environment variables: PYTHON is the path to the Python installation directory (for example, "C:\Python35") and TOXENV is the tox environment to use ("py35").

The problem is that tox has a built-in mapping from environment ("py35") to Python directory, and that mapping is wrong if we've had to install custom versions of Python in unusual places. For one thing, we install both 32- and 64-bit versions, in different directories, and Tox doesn't know where to find them.

So this line writes a file called "python3.5.bat" so that when Tox tries to run "python3.5", it will find it. The bat file simply has the actual path to the Python installation in it. The trick with this line was getting all of the escaping right: it's a YAML file containing a Windows command line which runs Python code to write a Windows bat file. "\x25" being the same as "%" definitely saved my sanity.

And getting code like this right is especially tricky because to run it on the CI system, you have to commit it and push it, and wait for the builds. It's like building a ship in a bottle: you can imagine the intricacy you need to create, and you can see the results of your efforts, but you have only a very tenuous set of tools to manipulate the process.

(In fact, as I write this, the Python 2.6 jobs are failing for both coverage.py and python-appveyor-demo, not sure why. It seems like the get-pip.py installation step is failing, but get-pip.py doesn't talk about what it is doing, so I'm not sure what's wrong. Back to the bottle...)

One of the nice things Appveyor does that some other CI systems don't is to hold onto build artifacts so that you can download them directly from Appveyor. This makes building wheels and kits there really convenient. I wrote a script to download all the artifacts from the latest build, so now it's really easy for me to include Windows runs in my coverage measurement, and I can build my own kits instead of having to ask other people to do it for me.

Along the way, I started another tool to help diagnose problems on remote machines: PyDoctor. (I know, there already is a pydoctor, I should probably change the name. Ideas welcome.)

After all the CI work, I feel like I have a vast distributed pinball machine. Every time I commit to Bitbucket:

  • documentation is built on Read The Docs
  • kicks off Windows builds on Appveyor
  • it's mirrored to GitHub, which then:
    • starts Linux builds on Travis
    • updates requirements on Requires.io
    • also starts a build on Circle CI because I wanted to compare it to Travis.

These services are incredibly useful, but keeping them configured and running smoothly is an art and an adventure in and of itself.

How many errors?

Monday 7 September 2015

A co-worker mentioned to me the other day that our test runner would run methods with the word "test" in the name, even if they didn't start with "test_". This surprised him and me, so I went looking into exactly how the function names were matched.

What I found was surprising: a single line of code with between three and five errors, depending on how you wanted to count them.

This is the line of code:

self.testMatch = re.compile(r'(?:^|[\\b_\\.%s-])[Tt]est' % os.sep)

Regexes are complicated, and it is easy to make mistakes. I'm not writing this to point fingers, or to label people as stupid. The point is that code is inherently complicated, and scrutiny and review are very very important for keeping things working.

The regex here is using an r"" string, as all regexes should. But notice there are two instances of double backslashes. The r"" string means that the regex will actually have two double backslashes. Each of them therefore means, "match a backslash." So we have a character class (in the square brackets, also called character ranges) with backslash in it twice. That's needless, one is enough.

But looking closer, what's that "b" doing in there? It will actually match a "b" character. Which means that "abtest" will match this pattern, but "bctest" will not. Surely that's a mistake.

Going back in the history of the repo, we see that the line used to be:

self.testMatch = re.compile(r'(?:^|[\b_\.%s-])[Tt]est' % os.sep)

That is, the backslashes used to be single rather than double. The doubling happened during a documentation pass: the docs needed the backslashes doubled, and I guess a misguided attempt at symmetry also doubled the backslashes in the code.

But with this older line, we can see that the intent of the backslashes was to get "\b" and "\." into the character set. The "\." wasn't necessary, a dot isn't special in a character set, so just "." would have been fine.

What's the "\b" for? The Python re docs say,

Matches the empty string, but only at the beginning or end of a word.

So the intent here was to force the word "test" to be at the boundary of a word. In which case, why include dot or dash in the regex? They would already define the boundary of a word.

But reading further in the re docs:

Inside a character range, \b represents the backspace character, for compatibility with Python’s string literals.

So the "\b" didn't match word boundaries at all, it matched a backspace character. I'm guessing it never encountered a backspace character in the directory, class, and function names it was being used on.

OK, so that probably explains how the dot and dash got there: the \b wasn't doing its job, and rather than get to the bottom of it, a developer threw in the explicit characters needed.

Let's look at os.sep. That's the platform-specific pathname separator: slash on Unix-like systems, backslash on Windows. String formatting is being used to insert it into the character class, so that the pathname separator will also be OK before the word "test". (Remember: if the \b had worked, we wouldn't need this at all.)

Look more closely at the result of that string formatting though: on Windows, the %s will be replaced with a backslash. The character class will then end with ".\-]". In a character class, backslashes still serve their escaping function, so this is the same as ".-]". That is, the backslash won't be used as a literal backslash at all. On Windows, this os.sep interpolation was pointless. (Keep in mind: this problem was solved when the incorrect backslash doubling happened.)

What's our final tally? I don't know, depends how you count. The line still has bugs ("abtest" vs "bctest"), and a long and checkered past.

Think carefully about what you write. Review it well. Get others to help. If something doesn't seem to be working, figure out why. Be careful out there.

« | » Main « | »