Appveyor

Monday 14 September 2015

I’ve just done a bunch of work on continuous integration for coverage.py. The biggest change is that now I’ve got Appveyor running Windows tests and building Windows kits.

Appveyor is a good service: the docs are helpful, and the support forum seems to be an obsession with the staff, especially Feodor Fitsner, who have always answered my questions within 12 hours regardless of when I ask them.

Oliver Grisel has a demonstration of building wheels on Appveyor which was very helpful in getting started.

Of course, part of the problem with supporting Windows is that it is unfamiliar to many of us. Appveyor provides the platform on which to run, but we still have to come up with the steps ourselves. And Python is a bit unfamiliar to Appveyor, so the steps include installing Python. It all gets a bit tangled.

The high point in my adventure came with this line:

install:
 - "python -c \"import os; open('python{0}.{1}.bat'.format(*os.environ['TOXENV'][2:]), 'w').write('@{0}\\\\python \\x25*\\n'.format(os.environ['PYTHON']))\""

Explanation: like most CI services, Appveyor is configured with a YAML file. This line is part of the install step before tests are run. It’s a Windows command line. Our appveyor.yml file installs a number of versions of Python, because Appveyor doesn’t have all the versions we need pre-installed. So each job sets two environment variables: PYTHON is the path to the Python installation directory (for example, “C:\Python35”) and TOXENV is the tox environment to use (“py35”).

The problem is that tox has a built-in mapping from environment (“py35”) to Python directory, and that mapping is wrong if we’ve had to install custom versions of Python in unusual places. For one thing, we install both 32- and 64-bit versions, in different directories, and Tox doesn’t know where to find them.

So this line writes a file called “python3.5.bat” so that when Tox tries to run “python3.5”, it will find it. The bat file simply has the actual path to the Python installation in it. The trick with this line was getting all of the escaping right: it’s a YAML file containing a Windows command line which runs Python code to write a Windows bat file. “\x25” being the same as “%” definitely saved my sanity.

And getting code like this right is especially tricky because to run it on the CI system, you have to commit it and push it, and wait for the builds. It’s like building a ship in a bottle: you can imagine the intricacy you need to create, and you can see the results of your efforts, but you have only a very tenuous set of tools to manipulate the process.

(In fact, as I write this, the Python 2.6 jobs are failing for both coverage.py and python-appveyor-demo, not sure why. It seems like the get-pip.py installation step is failing, but get-pip.py doesn’t talk about what it is doing, so I’m not sure what’s wrong. Back to the bottle...)

One of the nice things Appveyor does that some other CI systems don’t is to hold onto build artifacts so that you can download them directly from Appveyor. This makes building wheels and kits there really convenient. I wrote a script to download all the artifacts from the latest build, so now it’s really easy for me to include Windows runs in my coverage measurement, and I can build my own kits instead of having to ask other people to do it for me.

Along the way, I started another tool to help diagnose problems on remote machines: PyDoctor. (I know, there already is a pydoctor, I should probably change the name. Ideas welcome.)

After all the CI work, I feel like I have a vast distributed pinball machine. Every time I commit to Bitbucket:

  • documentation is built on Read The Docs
  • kicks off Windows builds on Appveyor
  • it’s mirrored to GitHub, which then:
    • starts Linux builds on Travis
    • updates requirements on Requires.io
    • also starts a build on Circle CI because I wanted to compare it to Travis.

These services are incredibly useful, but keeping them configured and running smoothly is an art and an adventure in and of itself.

Comments

[gravatar]
Ionel Cristian Mărieș 10:23 PM on 14 Sep 2015

I'm surprised you had the patience to change that PowerShell thing to support 3.5 - it ain't pretty at all. I gave up and just converted it to a python script: https://github.com/ionelmc/python-lazy-object-proxy/blob/master/ci/appveyor-bootstrap.py

Much more maintainable :-)

I believe you're handling the Tox configuration the hard way. Setting the bin path from the environment is way more straightforward, example:

* https://github.com/ionelmc/python-lazy-object-proxy/blob/master/appveyor.yml#L95-L106
* https://github.com/ionelmc/python-lazy-object-proxy/blob/master/tox.ini#L222

[gravatar]
Ned Batchelder 1:04 PM on 15 Sep 2015

@Ionel: you are right, we should use the Python script instead of the .ps1 script.

Not sure I like the tox technique, though: your tox.ini is complicated, and generated, which is a whole other layer of complexity. It would be easiest if I could use an environment variable to tell tox, "use this specific Python interpreter for this run."

[gravatar]
Ionel Cristian Mărieș 1:34 PM on 15 Sep 2015

This tox.ini has the same technique and it's not generated from a template: https://github.com/pytest-dev/pytest-cov/blob/master/tox.ini#L10-L16

The one in lazy-object-proxy is generated from a template because of the dependencies I want to test with, it's not really required for the {env:TOXPYTHON:...} trick.

Probably not worth switching coverage to run Tox like that but still, a better way if you start from a clean slate.

[gravatar]
Dominic Jodoin 2:58 AM on 24 Oct 2015

Great blog post Ned!

I'm curious to know how you do your mirroring from Bitbucket to Github?

Thanks!

Add a comment:

Ignore this:
Leave this empty:
Name is required. Either email or web are required. Email won't be displayed and I won't spam you. Your web site won't be indexed by search engines.
Don't put anything here:
Leave this empty:
URLs auto-link and some tags are allowed: <a><b><i><p><br><pre>.