Ad-hoc data breakpoints

Sunday 10 November 2013

A co-worker had a problem running a large test suite with nose. Modules were being imported from the wrong directory. Somehow, sys.path was having a “project/lib” directory stuffed into it, and we couldn’t figure out why. (tl;dr: it was nose’s fault, and we should have known about it, and it shouldn’t have been doing it in the first place.)

We searched our code for “sys.path.insert” and found more of them than we liked, but none of them accounted for the modification we were seeing. What we wanted was to run the tests in a debugger, with a data breakpoint set: stop when sys.path is modified.

Unfortunately, pdb doesn’t support breakpoints like that, maybe other debuggers do? So we whipped up an ad-hoc data breakpoint:

import pdb, sys

def trace(frame, event, arg):
    if sys.path[0].endswith("lib"):
        pdb.set_trace()
    return trace

sys.settrace(trace)

(Yes, it’s a little irksome that there are two different spellings of “set trace” there...)

A trace function is a Python function registered with the interpreter with sys.settrace(). This function will be called for every line of Python executed. Trace functions are the basis of debuggers, profilers, and code coverage tools.

Here we’ve written a very simple one: check to see if sys.path has been modified in the way we care about, and if so, break into the debugger. To be honest, I wasn’t quite sure what would happen if I tried to break into the debugger from inside a trace function, but when we ran the test suite with this code in place, it worked perfectly. We were dropped into the debugger just after nose added a “lib” directory to sys.path.

As it happens, nose tries to be helpful by adding a “src” and “lib” directory to the path, even though that’s an unusual layout for Python projects. Luckily, there’s a nose option to disable that bit of helpfulness, and our tests run just fine now.

If you find yourself in a similar situation, consider a simple trace function. It’s an advanced technique, but you don’t have to get too tricky, and can really tell you a lot about what your program is doing.

Comments

[gravatar]
Brandon Rhodes 11:40 PM on 10 Nov 2013

I had to switch away from nose years ago because of its path manipulation shenanigans (I ran into something that, if I recall, could not be turned off), and py.test has given me no trouble since my adoption of it as my framework. Turning on a similar trace function made my particular case take something like five minutes to finally find the problem because the trace function runs on every single line! That would be the only caveat that I would add. But, just as you claim, it did indeed work, and is invaluable when you have no other way to see what is ruining a setting or value.

[gravatar]
Brian Dant 4:29 AM on 11 Nov 2013

@brandon: I've been waffling between py.test and nose; glad to see your and Ned's anecdotes here. Do you use py.test primarily for the runner, or do you take advantage of other features? Dependency injection through funcargs? Bare asserts?

[gravatar]
Brandon Rhodes 7:59 PM on 11 Nov 2019

@Brian Dant: Only returning to this old post today do I see that you followed up with a question! For the record, I enjoy py.test’s discovery through its runner, its markers for skipping tests that do not apply to a particular platform or Python version, its bare asserts, and, yes, on some occasions for its magic functional argument mechanism for supplying fixtures. So, yes, I use all the features you mention, though the argument magic less and less often through the years.

I do have my own little testing framework "assay" that I think does things more cleanly and that has more advanced ideas in it, but as it’s kind of unfinished I only use it for one or two personal projects and otherwise I don’t inflict it on people :)

[gravatar]
Brian Dant 10:40 PM on 11 Nov 2019

Thanks, Brandon, for following up! I've gone the way of py.test over the last several years since these comments. Cool to know that you have a little testing framework.

Add a comment:

Ignore this:
Leave this empty:
Name is required. Either email or web are required. Email won't be displayed and I won't spam you. Your web site won't be indexed by search engines.
Don't put anything here:
Leave this empty:
URLs auto-link and some tags are allowed: <a><b><i><p><br><pre>.