Monday 17 December 2012 — This is almost 12 years old. Be careful.
Coverage.py has a C extension: the trace function is implemented in C for speed, since it’s invoked for every line of your program. This works great, and makes things much faster, but not every user has a C compiler properly installed on their machine. There’s also a Python implementation of the trace function, so we can make do without a compiler, but this make the installation a little tricky.
It turns out that distutils is great at attempting what you instruct it to, but not that great at letting you know what happened. In particular, building an extension can fail in a variety of ways. Separating actual build failures from other pathologies is not simple.
I had a way I was doing it, but still got bug reports from people who could not install the kit without getting tangled. Two suggestions were to do it the way SQLAlchemy did it, or the way simplejson did it. It turns out they do it the same way!
For example, compare SQLAlchemy’s setup.py:
ext_errors = (CCompilerError, DistutilsExecError, DistutilsPlatformError)
if sys.platform == 'win32' and sys.version_info > (2, 6):
# 2.6's distutils.msvc9compiler can raise an IOError when failing to
# find the compiler
ext_errors += (IOError,)
class BuildFailed(Exception):
def __init__(self):
self.cause = sys.exc_info()[1] # work around py 2/3 different syntax
class ve_build_ext(build_ext):
# This class allows C extension building to fail.
def run(self):
try:
build_ext.run(self)
except DistutilsPlatformError:
raise BuildFailed()
def build_extension(self, ext):
try:
build_ext.build_extension(self, ext)
except ext_errors:
raise BuildFailed()
except ValueError:
# this can happen on Windows 64 bit, see Python issue 7511
if "'path'" in str(sys.exc_info()[1]): # works with both py 2/3
raise BuildFailed()
raise
with simplejson’s setup.py:
if sys.platform == 'win32' and sys.version_info > (2, 6):
# 2.6's distutils.msvc9compiler can raise an IOError when failing to
# find the compiler
ext_errors = (CCompilerError, DistutilsExecError, DistutilsPlatformError,
IOError)
else:
ext_errors = (CCompilerError, DistutilsExecError, DistutilsPlatformError)
class BuildFailed(Exception):
pass
class ve_build_ext(build_ext):
# This class allows C extension building to fail.
def run(self):
try:
build_ext.run(self)
except DistutilsPlatformError, x:
raise BuildFailed()
def build_extension(self, ext):
try:
build_ext.build_extension(self, ext)
except ext_errors, x:
raise BuildFailed()
The code is clearly copied from the same source, even the name ve_build_ext is the same, as is its comment. SQLAlchemy has a new twist based on Python bug 7511, but it’s clear we’re all copying and pasting the same solution.
In fact, a Google search for that comment reveals a dozen or so packages using the same technique, with varying degrees of automatic fallback.
Does anyone know where this code first came from? What does the “ve” stand for in “ve_build_ext”? Is any of the work on better packaging tackling this issue? And how do people test this code? One suggestion was mocking out distutils, but the central issue here is properly catching the diverse exceptions that might occur if the compile fails. Help!
Comments
I would be interested in how bento deals with the optional C extension scenario. In general, I'm quite excited about bento, as the author has no shortage of cross-platform C (and Fortran!) extension experience. He was the numpy distutils guy for years, building it on all supported platforms. As much as pure-Python packaging solutions are useful, they're not going to solve the packaging problems of the scientific community unless they can integrate with libraries written in C and Fortran.
Add a comment: