Maintenance hatches

Saturday 3 December 2011This is over 11 years old. Be careful.

Here’s a pattern I’ve repeated too many times to count: build some software, deploy the software, software doesn’t work, wish I had more data points to figure out why. This is typical, and is often remedied after the fact by adding more logging. Everyone knows: you want tons of logging.

There’s another tool that I don’t use enough until I need it: maintenance hatches. On a physical machine, you need to be able to get at the inner workings of the thing to observe it, fiddle with it, and so on. The same is true for your software.

On a web site, these hatches take the form of URLs intended only for the developers and maintainers of the site to use. As a simple example, do you have a way to test the error handling on your live production server? If you want to see what happens when an exception occurs, you need a way to raise an exception on your live site. But you’ve tried hard to make sure that never happens in your code. Here’s a view that will do it for you:

def raise_error(request):
    """Raise an exception.  How else will we know our stack traces work?"""
    msg = request.GET.get("msg", "Something bad happened! (on purpose)")
    raise Exception(msg)

When you visit this URL, it raises an exception, simple. The message defaults to something that indicates it was intentional, but for convenience, you can provide your own message as a parameter on the URL. I’ve made it available only to staff members so that it can’t become a nuisance doorbell, and so that search engines won’t accidentally trigger it.

Once this view is in place, you’ll have a maintenance hatch that lets you look directly at a small part of your complex machinery. There are lots of other diagnostic tools that are possible:

def send_email(request):
    """Send an email to test the mail-sending infrastructure."""
    msg = request.GET.get("msg", "Test of sending email")
        'The body also says "%s"' % msg, 
    return HttpResponse("An email was sent to %s" %

def dump_settings(request):
    """Dump all the settings to the log file.""""----- Django settings:")
    for a in dir(settings):
        if a.startswith('__'):
            continue"%s%r" % (a, getattr(settings, a)))
    return HttpResponse("Settings have been logged.")

As you get deeper into your product-specific code, you’ll get away from simple general views like this into things that will only be useful to you, which is why you have to build them yourself rather than finding an off-the-shelf application.

These examples are for Django, but the principle is the same for any software, it isn’t limited to web applications.

One more I’ve found very useful: spawn a Celery task, to figure out if that machinery is properly configured:

def task_ping(request):
    """Send a simple task to a worker queue."""
    msg = request.GET.get("msg", "Task ping!")
    return HttpResponse("Sent a task with message, '%s'" % msg)

def pingtask(msg):
    print "Ping: %s" % msg

Often these views are written as a reaction to a specific problem, and then are forgotten, but they can be useful tools in the trenches. Write them for keeps, and document them so your staff knows they’re at their disposal, and they’ll be useful to you in the future.


i love the name "maintenance hatch"... always had those, but never named them :)
Yeah, 'maintenance hatch' is a great name for this concept.

For my current web app, I've actually built an entire REST API as a maintenance hatch (using djangorestframework). What I'm building is really just a web frontend for a bunch of existing web services, so it doesn't really need it's *own* REST API. However, by building one, I've been able to just focus on the *data* that I need for each page in the real site, without having to figure out how to display it yet. Instead, the REST framework will happily convert my dictionaries into nicely rendered HTML pages, so I can browse around, checking each resource contains all the relevant information.

I can also run a lot of my automated tests against the JSON interfaces, rather than having to screen-scrape the HTML.

When it comes time to build the real HTML pages, I can be confident the underlying data is right and just focus on testing the templates and rendering.
Funny thing is, I've been building these things for years, and the term "maintenance hatch" didn't come to me until I sat down to write this post.

@Nick: you've gone one step further: you've refactored your entire web application for testability, very nice!
Twisted contains something called "manhole" [1], and my own Pyro recently got a little extension called Flame [2]. Both of these allow you to open a live python REPL inside the server. How's that for a maintenance hatch? (twisted's name for it, manhole, is a synonym even :-)

Ofcourse you can't access these via the normal web application interface, but still, if you can sneak in from the back alley of your server it could be very powerful to peek into the live application.

Celery has had remote debugging support for some time too:
You can set breakpoints, or send celeryd a signal to make it start a remote debug session.
It also comes with a "cry handler": if you send the USR1 signal to a celery process it
will log the stack trace of all active threads.

Hatches, there should be more!
I use an HTTP REPL whenever I can.

Add a comment:

Ignore this:
Leave this empty:
Name is required. Either email or web are required. Email won't be displayed and I won't spam you. Your web site won't be indexed by search engines.
Don't put anything here:
Leave this empty:
Comment text is Markdown.