Decorated fabric over the edge

Tuesday 10 January 2012

Like many, I use Fabric to write deploy procedures, but I feel like I'm doing it wrong. Fabric is fundamentally based on the ideas of "hosts" and "tasks". You write a Python file whose functions are tasks, and from the command line you can ask that a list of tasks be performed on a list of hosts.

Tasks can be decorated to affect their execution, for example, the @runs_once decorator will mean the function is only executed once, no matter how many hosts are specified. This can be useful for performing initial work, such as preparing a tarball to be copied to many hosts. So for example, I can write something like this:

@task
def deploy():
    make_tarball()
    copy_tarball()

@runs_once
def make_tarball()
    # .. create a .tgz ..

def copy_tarball()
    put('the_tar_ball.tgz', '/tmp/')

Fabric will run this by running the deploy task for each host, which will call both make_tarball and copy_tarball, but the runs_once decorator on make_tarball means that it will only be executed for the first host, while copy_tarball will be executed for all of them.

This is great, and building on it for a multi-server deploy, I wanted to have functions that would be run on a subset of the hosts. I have servers divided into roles: app server vs. static content server, for example. Fabric provides a role system, and includes a @roles decorator to control what gets run where:

env.roledefs.update({
    'app': ['www1', 'www2'],
    'static': ['stat1']
})

@roles('app')
def my_func():
    pass

But we run into a problem: @roles only works on top-level tasks invoked from the command line. If I decorate my copy_tarball function with it, it will be ignored. This is because of how the decorator has been written: it annotates the function with role information, and the Fabric main loop knows how to read that annotation to decide what tasks to run on which hosts.

I wanted a deploy script that looked something like this:

@task
def deploy():
    copy_to_apps()
    copy_to_static()

@only_roles('app')
def copy_to_apps():
    #.. copy stuff ..

@only_roles('static')
def copy_to_static():
    #.. copy stuff ..

So I wrote my own decorator to do roles the way I wanted:

def only_roles(*roles):
    """Make a function run only on hosts that have certain roles."""
    def _dec(fn):
        @functools.wraps(fn)
        def _wrapped(*args, **kwargs):
            for role in roles:
                if env.host_string in env.roledefs.get(role, ()):
                    return fn(*args, **kwargs)
        return _wrapped
    return _dec

But I felt funny about this: I saw something in the Fabric docs that sounded like just what I wanted, but it didn't work as I thought it would, so I had to write my own. This makes me think I'm using Fabric wrong.

The runs_once decorator is great for doing one-time initial work, and I found I wanted a book-end for it: a way to do one-time cleanup work. Fabric provided nothing, and I could see why: there's no global knowledge about all the hosts and tasks, and no way to specify work to be done after they are through. For that matter, there's no way to specify work to be done before they start, but @runs_once provides that effect.

So I wrote another decorator, this one more devious and risky:

def runs_last(func):
    """A decorator to run the function only on the *last* host.

    This only works if you don't apply any other restrictions
    on the function.

    """
    func.times_invoked = 0
    @functools.wraps(func)
    def decorated(*args, **kwargs):
        func.times_invoked += 1
        all_hosts = set()
        for hosts in env.roledefs.values():
            all_hosts.update(hosts)
        if func.times_invoked == len(all_hosts):
            func(*args, **kwargs)
    return decorated

Here we count the number of invocations, and guess at the number of the last one based on the hosts we know about. There are a variety of ways this could not work, but it was fine in my environment.

Since I'm sharing useful Fabric decorators, here's one that prevents repetitive work being done that won't have any extra effect:

def idempotent(func):
    """Don't invoke `func` more than once for host and arguments."""
    func.past_results = {}
    @functools.wraps(func)
    def decorated(*args, **kwargs):
        key = (env.host_string, args, tuple(kwargs.items()))
        if key not in func.past_results:
            func.past_results[key] = func(*args, **kwargs)
        return func.past_results[key]
    return decorated

Am I using Fabric wrong? It seems like maybe I'm expecting it to do too much, like the right way is to have my deploy() function in a larger script somehow. Or is Fabric fine like this, and I've just missed the right path?

tagged: » 6 reactions

Comments

[gravatar]
Jeff 1:55 PM on 10 Jan 2012

You might want to see if execute() works for you; it was written partly to help a lot of these non-base use cases.

http://docs.fabfile.org/en/1.3.3/usage/execution.html#execute

[gravatar]
Matti Mäki 2:59 PM on 10 Jan 2012

You might also be interested of reading about Poni which is a system configuration tool written in Python:
http://melor.github.com/poni/

[gravatar]
Ned Batchelder 3:35 PM on 10 Jan 2012

@Jeff: Thanks! execute() seems to be just the thing. The doc example is just like mine! Fabric 1.3 came out after I started down this path, and I missed it.

[gravatar]
Jeff 3:49 PM on 10 Jan 2012

@Ned: Glad to hear it, that's what I suspected (1.3 being newer than your efforts). Apologies for not having the time to more deeply dissect your examples and show how execute() could help -- busy day at work :)

[gravatar]
Ned Batchelder 4:08 PM on 10 Jan 2012

@Jeff, please, no apologies necessary: you pointed me directly at the right answer!

[gravatar]
Malcolm Tredinnick 4:16 PM on 10 Jan 2012

The "book-ending" analogy you used for wanting clean up is an API design flaw that's common. It's usually not particularly severe, particularly in small-impact code, which is why it rarely makes top-10 lists for many people and makes it a flaw not a showstopper. But the longer the API exists and more broadly it gets used, the more you start to notice the omission. Separating interfaces -- at the class level, at the module level and at the library level -- into set up, actions and clean up and trying to make setting up and cleaning up be symmetrical without being too hard to use is a sign of expert level design to me.

Add a comment:

name
email
Ignore this:
not displayed and no spam.
Leave this empty:
www
not searched.
 
Name and either email or www are required.
Don't put anything here:
Leave this empty:
URLs auto-link and some tags are allowed: <a><b><i><p><br><pre>.