Ad-hoc data breakpoints

Sunday 10 November 2013

A co-worker had a problem running a large test suite with nose. Modules were being imported from the wrong directory. Somehow, sys.path was having a "project/lib" directory stuffed into it, and we couldn't figure out why. (tl;dr: it was nose's fault, and we should have known about it, and it shouldn't have been doing it in the first place.)

We searched our code for "sys.path.insert" and found more of them than we liked, but none of them accounted for the modification we were seeing. What we wanted was to run the tests in a debugger, with a data breakpoint set: stop when sys.path is modified.

Unfortunately, pdb doesn't support breakpoints like that, maybe other debuggers do? So we whipped up an ad-hoc data breakpoint:

import pdb, sys

def trace(frame, event, arg):
    if sys.path[0].endswith("lib"):
        pdb.set_trace()
    return trace

sys.settrace(trace)

(Yes, it's a little irksome that there are two different spellings of "set trace" there...)

A trace function is a Python function registered with the interpreter with sys.settrace(). This function will be called for every line of Python executed. Trace functions are the basis of debuggers, profilers, and code coverage tools.

Here we've written a very simple one: check to see if sys.path has been modified in the way we care about, and if so, break into the debugger. To be honest, I wasn't quite sure what would happen if I tried to break into the debugger from inside a trace function, but when we ran the test suite with this code in place, it worked perfectly. We were dropped into the debugger just after nose added a "lib" directory to sys.path.

As it happens, nose tries to be helpful by adding a "src" and "lib" directory to the path, even though that's an unusual layout for Python projects. Luckily, there's a nose option to disable that bit of helpfulness, and our tests run just fine now.

If you find yourself in a similar situation, consider a simple trace function. It's an advanced technique, but you don't have to get too tricky, and can really tell you a lot about what your program is doing.

Comments

[gravatar]
Brandon Rhodes 11:40 PM on 10 Nov 2013

I had to switch away from nose years ago because of its path manipulation shenanigans (I ran into something that, if I recall, could not be turned off), and py.test has given me no trouble since my adoption of it as my framework. Turning on a similar trace function made my particular case take something like five minutes to finally find the problem because the trace function runs on every single line! That would be the only caveat that I would add. But, just as you claim, it did indeed work, and is invaluable when you have no other way to see what is ruining a setting or value.

[gravatar]
Brian Dant 4:29 AM on 11 Nov 2013

@brandon: I've been waffling between py.test and nose; glad to see your and Ned's anecdotes here. Do you use py.test primarily for the runner, or do you take advantage of other features? Dependency injection through funcargs? Bare asserts?

Add a comment:

name
email
Ignore this:
not displayed and no spam.
Leave this empty:
www
not searched.
 
Name and either email or www are required.
Don't put anything here:
Leave this empty:
URLs auto-link and some tags are allowed: <a><b><i><p><br><pre>.