Skipping C extensions

Monday 17 December 2012This is almost 12 years old. Be careful.

Coverage.py has a C extension: the trace function is implemented in C for speed, since it’s invoked for every line of your program. This works great, and makes things much faster, but not every user has a C compiler properly installed on their machine. There’s also a Python implementation of the trace function, so we can make do without a compiler, but this make the installation a little tricky.

It turns out that distutils is great at attempting what you instruct it to, but not that great at letting you know what happened. In particular, building an extension can fail in a variety of ways. Separating actual build failures from other pathologies is not simple.

I had a way I was doing it, but still got bug reports from people who could not install the kit without getting tangled. Two suggestions were to do it the way SQLAlchemy did it, or the way simplejson did it. It turns out they do it the same way!

For example, compare SQLAlchemy’s setup.py:

ext_errors = (CCompilerError, DistutilsExecError, DistutilsPlatformError)
if sys.platform == 'win32' and sys.version_info > (2, 6):
    # 2.6's distutils.msvc9compiler can raise an IOError when failing to
    # find the compiler
    ext_errors += (IOError,)

class BuildFailed(Exception):

    def __init__(self):
        self.cause = sys.exc_info()[1]  # work around py 2/3 different syntax

class ve_build_ext(build_ext):
    # This class allows C extension building to fail.

    def run(self):
        try:
            build_ext.run(self)
        except DistutilsPlatformError:
            raise BuildFailed()

    def build_extension(self, ext):
        try:
            build_ext.build_extension(self, ext)
        except ext_errors:
            raise BuildFailed()
        except ValueError:
            # this can happen on Windows 64 bit, see Python issue 7511
            if "'path'" in str(sys.exc_info()[1]):  # works with both py 2/3
                raise BuildFailed()
            raise

with simplejson’s setup.py:

if sys.platform == 'win32' and sys.version_info > (2, 6):
   # 2.6's distutils.msvc9compiler can raise an IOError when failing to
   # find the compiler
   ext_errors = (CCompilerError, DistutilsExecError, DistutilsPlatformError,
                 IOError)
else:
   ext_errors = (CCompilerError, DistutilsExecError, DistutilsPlatformError)

class BuildFailed(Exception):
    pass

class ve_build_ext(build_ext):
    # This class allows C extension building to fail.

    def run(self):
        try:
            build_ext.run(self)
        except DistutilsPlatformError, x:
            raise BuildFailed()

    def build_extension(self, ext):
        try:
            build_ext.build_extension(self, ext)
        except ext_errors, x:
            raise BuildFailed()

The code is clearly copied from the same source, even the name ve_build_ext is the same, as is its comment. SQLAlchemy has a new twist based on Python bug 7511, but it’s clear we’re all copying and pasting the same solution.

In fact, a Google search for that comment reveals a dozen or so packages using the same technique, with varying degrees of automatic fallback.

Does anyone know where this code first came from? What does the “ve” stand for in “ve_build_ext”? Is any of the work on better packaging tackling this issue? And how do people test this code? One suggestion was mocking out distutils, but the central issue here is properly catching the diverse exceptions that might occur if the compile fails. Help!

Comments

[gravatar]
FIW you can also pass optional=True to your extension, and have distutils ignore compilation error (2.6+)
[gravatar]
@tarek: that's a very useful thing to know! Is there a reason it isn't documented? (http://docs.python.org/2/distutils/apiref.html#distutils.core.Extension)
[gravatar]
Nope, that's a miss we should fix.
[gravatar]
ve_build_ext has been in VisionEgg (http://visionegg.org) since 2003: https://github.com/visionegg/visionegg/commit/e273447c8aa0601055646a1a2d23c118c9579818 , and the project's name is a likely explanation for the ve_ prefix. Hence, credit should probably go to VisionEgg's author Andrew Straw.
[gravatar]
@Marcel: thanks! That's definitely the source of the name and the comment. Interestingly, the technique itself seems to have morphed significantly since then...
[gravatar]
Unfortunately "optional=True" flag is not available in Python 2.x: http://bugs.python.org/issue5583 (I just checked this: there is indeed no "optional" flag handling code in distutils shipped with Python 2.7). The "optional" flag is documented in Python 3.x docs as it should.
[gravatar]
Oh jeez you are right I forgot about the revert.. :(
[gravatar]
When I added this to the Vision Egg in 2003, I based this off some other code, although I no longer remember what. Perhaps it was something in PyOpenGL or pygame. Definitely the fundamental idea wasn't mine. So, I'm sorry your search will have to go on longer. :)

I would be interested in how bento deals with the optional C extension scenario. In general, I'm quite excited about bento, as the author has no shortage of cross-platform C (and Fortran!) extension experience. He was the numpy distutils guy for years, building it on all supported platforms. As much as pure-Python packaging solutions are useful, they're not going to solve the packaging problems of the scientific community unless they can integrate with libraries written in C and Fortran.
[gravatar]
This wouldn't have come from Pygame around 2003. Back then Pygame used a separate config file to specify which subsystems should be built, but then required those to work. (I think)
[gravatar]
Bob Ippolito must have copied it from Vision Egg to simplejson, I then copied it from there to SQLAlchemy and at least one other place (Jinja2 IIRC). I was wondering what ve_ meant!

Add a comment:

Ignore this:
Leave this empty:
Name is required. Either email or web are required. Email won't be displayed and I won't spam you. Your web site won't be indexed by search engines.
Don't put anything here:
Leave this empty:
Comment text is Markdown.