Maintenance hatches

Saturday 3 December 2011

Here's a pattern I've repeated too many times to count: build some software, deploy the software, software doesn't work, wish I had more data points to figure out why. This is typical, and is often remedied after the fact by adding more logging. Everyone knows: you want tons of logging.

There's another tool that I don't use enough until I need it: maintenance hatches. On a physical machine, you need to be able to get at the inner workings of the thing to observe it, fiddle with it, and so on. The same is true for your software.

On a web site, these hatches take the form of URLs intended only for the developers and maintainers of the site to use. As a simple example, do you have a way to test the error handling on your live production server? If you want to see what happens when an exception occurs, you need a way to raise an exception on your live site. But you've tried hard to make sure that never happens in your code. Here's a view that will do it for you:

@staff_member_required
def raise_error(request):
    """Raise an exception.  How else will we know our stack traces work?"""
    msg = request.GET.get("msg", "Something bad happened! (on purpose)")
    raise Exception(msg)

When you visit this URL, it raises an exception, simple. The message defaults to something that indicates it was intentional, but for convenience, you can provide your own message as a parameter on the URL. I've made it available only to staff members so that it can't become a nuisance doorbell, and so that search engines won't accidentally trigger it.

Once this view is in place, you'll have a maintenance hatch that lets you look directly at a small part of your complex machinery. There are lots of other diagnostic tools that are possible:

@staff_member_required
def send_email(request):
    """Send an email to test the mail-sending infrastructure."""
    msg = request.GET.get("msg", "Test of sending email")
    send_mail(
        msg, 
        'The body also says "%s"' % msg, 
        settings.DEFAULT_FROM_EMAIL, 
        [request.user.email], 
        fail_silently=False
        )
    return HttpResponse("An email was sent to %s" % request.user.email)

@staff_member_required
def dump_settings(request):
    """Dump all the settings to the log file."""
    log.info("----- Django settings:")
    for a in dir(settings):
        if a.startswith('__'):
            continue
        log.info("%s%r" % (a, getattr(settings, a)))
    return HttpResponse("Settings have been logged.")

As you get deeper into your product-specific code, you'll get away from simple general views like this into things that will only be useful to you, which is why you have to build them yourself rather than finding an off-the-shelf application.

These examples are for Django, but the principle is the same for any software, it isn't limited to web applications.

One more I've found very useful: spawn a Celery task, to figure out if that machinery is properly configured:

@staff_member_required
def task_ping(request):
    """Send a simple task to a worker queue."""
    msg = request.GET.get("msg", "Task ping!")
    pingtask.delay(msg)
    return HttpResponse("Sent a task with message, '%s'" % msg)

@task
def pingtask(msg):
    print "Ping: %s" % msg

Often these views are written as a reaction to a specific problem, and then are forgotten, but they can be useful tools in the trenches. Write them for keeps, and document them so your staff knows they're at their disposal, and they'll be useful to you in the future.

tagged: » 6 reactions

Comments

[gravatar]
noam 9:44 AM on 3 Dec 2011

i love the name "maintenance hatch"... always had those, but never named them :)

[gravatar]
Nick Coghlan 10:36 AM on 3 Dec 2011

Yeah, 'maintenance hatch' is a great name for this concept.

For my current web app, I've actually built an entire REST API as a maintenance hatch (using djangorestframework). What I'm building is really just a web frontend for a bunch of existing web services, so it doesn't really need it's *own* REST API. However, by building one, I've been able to just focus on the *data* that I need for each page in the real site, without having to figure out how to display it yet. Instead, the REST framework will happily convert my dictionaries into nicely rendered HTML pages, so I can browse around, checking each resource contains all the relevant information.

I can also run a lot of my automated tests against the JSON interfaces, rather than having to screen-scrape the HTML.

When it comes time to build the real HTML pages, I can be confident the underlying data is right and just focus on testing the templates and rendering.

[gravatar]
Ned Batchelder 11:36 AM on 3 Dec 2011

Funny thing is, I've been building these things for years, and the term "maintenance hatch" didn't come to me until I sat down to write this post.

@Nick: you've gone one step further: you've refactored your entire web application for testability, very nice!

[gravatar]
Irmen de Jong 3:26 PM on 3 Dec 2011

Twisted contains something called "manhole" [1], and my own Pyro recently got a little extension called Flame [2]. Both of these allow you to open a live python REPL inside the server. How's that for a maintenance hatch? (twisted's name for it, manhole, is a synonym even :-)

Ofcourse you can't access these via the normal web application interface, but still, if you can sneak in from the back alley of your server it could be very powerful to peek into the live application.

[1] http://www.lothar.com/tech/twisted/manhole.xhtml
[2] http://packages.python.org/Pyro4/flame.html

[gravatar]
Ask Solem 5:19 AM on 4 Dec 2011

Celery has had remote debugging support for some time too: http://ask.github.com/celery/reference/celery.contrib.rdb.html
You can set breakpoints, or send celeryd a signal to make it start a remote debug session.
It also comes with a "cry handler": if you send the USR1 signal to a celery process it
will log the stack trace of all active threads.

Hatches, there should be more!

[gravatar]
Robert Brewer 9:45 AM on 4 Dec 2011

I use an HTTP REPL whenever I can. http://www.aminus.net/wiki/HTTPREPL

Add a comment:

name
email
Ignore this:
not displayed and no spam.
Leave this empty:
www
not searched.
 
Name and either email or www are required.
Don't put anything here:
Leave this empty:
URLs auto-link and some tags are allowed: <a><b><i><p><br><pre>.