There's a lot of talk these days about new web frameworks. Django, Turbogears and Ruby on Rails are leading the charge on new ways to build web applications. They undoubtably save time, improve productivity, and make developers happier. It's easier to build sites with these frameworks and in these languages than with competing technologies.

Of course, all of the interest and hype will result in some backlash. Hacknot has an amusing rant, All Aboard the Gravy Train (with a great pun, "Now departing platform one"). He ultimately dismisses the new frameworks with:

Your time and energy is better invested in improving your abilities and skills than in adding another notch to your technology belt.

This is throwing out the baby with the bath water, as Pete Lyons ably argues in Defending Ruby on Rails:

The problem with these assertions are that they assume that learning the new technology neither offers new insight into the 'principles or techniques of software development' nor would 'improve your ability and skill'.

I've been working in Django for eight months now, and it really is faster and better than other ways we could have chosen to build Tabblo. Django and Python have made the job easier, but only the actual writing of the software, which in my mind, is the easy part to begin with.

Building a great product involves a lot of hard work, much of which has nothing to do with coding:

• What should the product do?
• Who are the customers? What do they care about?
• Of our two (three?, n?) conflicting goals, which should I work on today?
• How do we architect the software to anticipate the unseen future?
• What's the right architecture to ensure the application performs well?
• What can we do to make sure the site stays running around the clock?
• How do we balance short-term vs. long-term needs?
• Which add-on libraries and tools are right to incorporate into the product?
• How do we assess the quality of the product?
• What's the right balance between adding features and fixing bugs?
• Which are the bugs that really interfere with the product's success?
• When is the software "done enough"?
• How do we gauge the risk of any decision we make? How do we mitigate it? What's plan B?
• What are the lessons from previous projects that apply here? Is this history repeating itself, or are we in new territory?

And this isn't even covering the larger organizational issues of management, team building, and general inter-personal stuff that any software engineer on a real product will have to deal with, not to mention whole-business concerns like marketing, sales, support, and so on.

When I worked at Digital in the printer group, there was a joke among the software engineers: we claimed the hardware guys looked down on us because they thought that writing software was "just typing". By focusing purely on coding issues, we software engineers run the risk of taking the same stupid view of our world.

At a certain simplisitic level, a web app is a straightforward piece of software: it has to accept request at URLs, dive into a database to get some data, build some results, format them into HTML, and return them to the browser. Django and Rails make this process easier, absolutely. But building a great web application involves so much more difficult work. The frameworks can help by simplifying coding so that we can focus on the remaining difficult problems.

When the hype-masters claim that their frameworks and languages make coding an application easier, they are absolutely right. But that's already the easy part.

A commenter on the White House photo post mentioned that he'd never seen a photo of me, and it's true, this site is devoid of self-images. To help correct that, here are some more, but instead of a tux, how about overalls?

In 1980, I was a senior at the Bronx High School of Science. I was also on the math team. For reasons I don't remember, a Life magazine photographer came to photograph us. The photos were never used in the magazine, but my dad was friends with someone in the photo department there, and got some rough prints. Here they are:

Last week the photo of us at the White House finally arrived in the mail. It's the two of us, the President and Mrs. Bush and Eunice Kennedy Shriver. They included two hardcopy prints, one signed and inscribed:

I've updated coverage.py with a number of (mostly user-contributed) changes:

• Function decorators no longer confuse it (thanks, Joseph Tate).
• Fixes for argument handling (thanks, Mark van der Wal and Sigve Tjora).
• A new parallel mode lets many processes collect coverage data at once, and then later collate everything together for unified reporting (thanks, Geoff Bache).

Earlier this month, Ruby on Rails reached a milestone: they issued a security advisory:

This is a MANDATORY upgrade for anyone not running on a very recent edge (which isn't affected by this). If you have a public Rails site, you MUST upgrade to Rails 1.1.5. The security issue is severe and you do not want to be caught unpatched.

The issue is in fact of such a criticality that we're not going to dig into the specifics. No need to arm would-be assalients.

24 hours later, they explained the whole situation. Unfortunately, it turned out that the mandatory 1.1.5 patch not only was not sufficient, requiring a 1.1.6 patch, but if you had been running 1.0 or 1.1.3, then upgrading to 1.1.5 made your system less secure!

This naturally engendered a great deal of discussion on the Ruby blog. As you may have guessed, the opinions there run the gamut. On the plus side, Matt Van Dusen said:

As far as I can tell, the Ruby team did what any good development group would do if they had a potentially crippling problem in their fundamental code: Rolled out a fix as fast as they could and warning their users.

Compare that to Microsoft, where it takes months to get ANY kind of disclosure, and it takes a massive attack for them to admit they f*cked up their code, and to have them tell us it's gonna be at least 2 weeks before they could possibly fix it - thanks guys, you've just told the entire cracker community how to break any Windoze box in the world, and no fixes for 2 weeks?

and Jon Shea said:

I don't understand any of this griping. I think the Rails team's response was flawless. I can't think of even a single example of a commercial software package's security flaw being handled as well.

DHH et al., thanks a million times over for all your hard work. Rails is an inspired piece of software.

On the other hand, "Upset Commercial Rails User" said:

I understand this is an open source project, but if you want to continue to see the adoption grow, especially by commercial users, do us a favor and be more thorough before you rush out to tell the world that disaster is moments away.

It's bad enough to deal with this type of thing from Microsoft. Now I have to spend time and resources for this update as well.

Get your act together, I'm sure you guys have professional development or IT jobs as well, and one massive deployment to fix a problem that should have been solved in yesterday's massive deployment should upset you as well.

I'm not concerned that there was a security problem in Rails, that happens. What bothers me is the haste in which the first "fix" was rushed.

to which DHH (the creator of Rails) unfortunately replied:

I'm puzzled, what's a commercial Rails user? I don't recall there being a sticker price on the box. We care equally for all users since everyone is paying the same price: zilch.

Okay, that's not true. We actually care more about users who don't feel the need to hide behind aliases like "Commercial Rails user".

If you have something to say, please stand by your words by signing it with your real name. Anything else is a surefire way to get anything you have to say discounted as trolling.

The tone of the discussion became a bit more controversial. One of the last comments (by Tom Barrick) is:

DHH: Grow up.

As I said, it's a milestone for Ruby on Rails. They've had a pheneomenal success, a huge adoption rate, and tremendous press. This security patch was a real-world hiccup. All software has security issues and bugs. What determines success is how the team deals with them.

I think the 24 hour turnaround was very good, and the full (eventual) disclosure was also very good. Keeping the vulnerability secret for a day was bad, especially since this is an open-source project, so the details were publically available, but only to those able to diff and grok the code. The inadvertent lowering of security with a mandatory patch was very bad, as was the sniping at your customers in the blog comments. Growing pains all around.

I'm not a Rails user, but I'm hoping they weather this storm well. I'd like projects like Rails to succeed and make inroads into the big guys' markets.

These days, everything on line can be monitored for other people's attention. Web sites have referrers, blog posts have Technorati statistics. We know how many people comment on our posts, our books at Amazon have sales ranks. We search for our names on Google, look up our house value on Zillow. At Tabblo we show you statistics about how many people have joined the site due to your invitations. As much as possible, the web gives us quantitative ways to measure other people's interest in our world.

Last night I had vivid and detailed dreams involving two people: Bill Engvall was performing a controversal juggling routine, and my high school friend Bruce Burns and I were trying to meet up with a bunch of people on a train.

But Bill and Bruce don't know I was dreaming about them. Somehow in our ultra-linked and richly annotated world, there ought to be a way for us to check each morning to see how many dreams there were about us the night before. What a fascinating statistic that would be! How much more insight we would have into our effect on the world...

I was swimming in the ocean today, the highlight of my vacation days. After about 20 minutes, the lifeguard (and my frantic wife) waved me out of the water because there was a school of bluefish feeding nearby, unbeknownst to me. I was heading toward the frenzy, and I would have been, in the words of the lifeguard, "part of the food chain". It was an exciting moment in an otherwise relaxed vacation week, but it made me think of something that happened on the same beach a year ago.

» Read the whole harrowing tale... (12 paragraphs)

Working in a startup, you are always aware of possible "exit strategies": ways that your company's startup phase could end. Good ones are acquisition and IPO. Bad ones include running out of money. Well-regarded calendaring site Kiko has a new one: auction the company on eBay.

They've built an impressive site. Their reasons for selling it seem a little capricious:

We are selling Kiko because we want to have time to work on other projects as a development team. We had a project in mind we just didn't want to wait on :)

They may have a problem convincing future investors that they are in this for the long run, but if it works for them...

This is one of those times when scientists are finally doing important work, the kind of thinking and research that could make a difference to ordinary people in their everyday lives: Michael LaBarbera has written a detailed examination of The Biology of B-Movie Monsters. He carefully considers the engineering principles underlying the structure of bodies, and works out just what would happen (for example) if you shrunk a man to an inch tall:

When the Incredible Shrinking Man stops shrinking, he is about an inch tall, down by a factor of about 70 in linear dimensions. Thus, the surface area of his body, through which he loses heat, has decreased by a factor of 70 × 70 or about 5,000 times, but the mass of his body, which generates the heat, has decreased by 70 × 70 × 70 or 350,000 times. He's clearly going to have a hard time maintaining his body temperature (even though his clothes are now conveniently shrinking with him) unless his metabolic rate increases drastically.

Luckily, his lung area has only decreased by 5,000-fold, so he can get the relatively larger supply of oxygen he needs, but he's going to have to supply his body with much more fuel; like a shrew, he'll probably have to eat his own weight daily just to stay alive. He'll also have to give up sleeping and eat 24 hours a day or risk starving before he wakes up in the morning (unless he can learn the trick used by hummingbirds of lowering their body temperatures while they sleep).

Because of these relatively larger surface areas, he'll be losing water at a proportionally larger rate, so he'll have to drink a lot, too. We see him drink once in the movie—he dips his hand into a puddle and sips from his cupped palm. The image is unremarkable and natural, but unfortunately wrong for his dimensions: at his size surface tension becomes a force comparable to gravity. More likely, he'd immerse his hand in the pool and withdraw it coated with a drop of water the size of his head. When he put his lips to the drop, the surface tension would force the drop down his throat whether or not he chooses to swallow.

As an engineer, it's fascinating to read this sort of analysis. Biological creatures are by necessity taken as a given by most of us. The question of how a one-inch man would work simply doesn't apply. But we do this type of analysis on systems all the time, so it's especially interesting to see it applied outside our domain.

At work, my ssh session to our servers time out quite frequently. It used to bother me a lot because it meant having to login to the machines many times a day, and losing the on-screen history of what I had been doing.

Not anymore. A co-worker introduced me to screen, a terminal multiplexor. In a nutshell, it manages virtual terminals that persist even when you log out. So if a connection to a server is dropped, logging in again will re-connect you to the session you had been working in.

Screen manages many virtual terminals, so one OS window can contain many Linux sessions within it, reducing desktop clutter.

Screen is quite complex, in the emacs mold: it can do many things, and has many commands, some of which are bound to keystrokes, but many of which are not. There are esoteric escape sequences that can be used to hyper-customize the environment, and more options than I can possibly understand. If you find yourself using many console windows on Unix boxes, do yourself a favor and learn enough about screen to get it working for you.

The documentation out there tends to be a bit overwhelming as well, falling into two camps: the exhaustive reference manual that begins with the command-line switches, or the informal tutorial that starts out talking about stuff you don't understand. Here's a shorter getting-started guide that might be helpful.

I'm tempted to write a mini-tutorial myself, but I don't think I could really improve on what's out there. I will give you my .screenrc file:

# Ned's .screenrc
escape ^L^L
vbell off
defscrollback 8000
hardstatus alwayslastline
hardstatus string '%{= KW}%-w%{= gW}%50> %n %t %{= KW}%+w%< %{= kG}%-=%{-}'

What does this do?

• Sets the escape character to ^L, because I don't need that for any real use, and the default ^A is too handy for command-line editing.
• Turns off the visible bell, which is too much flashing for me.
• Sets a scrollback buffer of 8000 lines, because I like to look back through my console window for old stuff. One thing about screen: it manages the console window such that your real console window never scrolls, so you have to change where you store your off-screen lines. Your true console window doesn't need them anymore; "defscrollback 8000" configures screen to keep them for you.
• Sets the hard status line to be the last line in the window, and to give it a simple tabbed look. This makes the last line in the window a list of your current screen sessions, in a nice green-on-gray theme.

Screen is a god-send.

Washington and New York states both recently upheld their bans against gay marriage, and with very similar logic. But the reasoning is pretty flimsy.

Basically, both states' courts said the legislatures could ban gay marriage because the state has an interest in ensuring procreation. This is a laughable reason to preclude state recognition of stable relationships.

First of all, plenty of heterosexual couples get married without procreating. When two 90-year-olds get married, it's considered a heart-warming event that makes it onto the local news. Will states begin refusing marriage licenses to infertile couples, arguing that only by reserving marriage for the fertile can they guarantee the continuity of humanity? It's absurd.

As Ellen Goodman wrote in today's Boston Globe:

If marriage is for procreation, shouldn't they refuse to wed anyone past menopause? Shouldn't they withhold a license, let alone blessings and benefits, from anyone who is infertile? What about those who choose to be childless? Nothing borrowed or blue for them. Indeed the state could offer young couples licenses with sunset clauses. After five years they have to put up (kids) or split up.

Of course the states' other interest is in families "headed by the children's biological parents." Why then give licenses to the couples who are raising 1.5 million adopted children? We can ban those blended families like, say, the Brady Bunch. And surely we should release partners from their vows upon delivery of their offspring to the nearest college campus.

This is where the courts' reasoning leads us, and I use the word "reasoning" loosely. If anything, these two decisions are proof that the courts and the country are running out of reasons for treating straight and gay citizens differently.

Second, many gay couples intend to raise children, and are in fact allowed to do so by both New York and Washington state. Are we to believe that it is good to raise adopted children, but married parents are not important for those kids?

As Dan Savage said in the New York Times:

These defeats have demoralized supporters of gay marriage, but I see a silver lining. If heterosexual instability and the link between heterosexual sex and human reproduction are the best arguments opponents of same-sex marriage can muster, I can't help but feel that our side must be winning. Insulting heterosexuals and discriminating against children with same-sex parents may score the other side a few runs, but these strategies won't win the game.

I think it is inevitable that society will come around to recognizing gay marriage, just as 50 years ago we recognized inter-racial marriage. It's only a matter of time.

Dave Shea points to a CNET article that says IE7 will be a high priority update, meaning many many people will upgrade to IE7 automatically. In other words, the adoption rate will be very fast. Does your app work with it?

This is one of those debugging tales that reveals multiple layers of helpful magic, all of which combine to make things not work, and keep you from figuring out why.

I was experimenting with a complex query in Django 0.91. I needed to join through a number of tables, and the last table needed to use an IN expression to get results that applied to a number of different users at once. The code was something along the lines of:

idlist = ','.join([str(i) for i in ids])
tlist = things.get_list(
                        tables=['middle','people'],
                        where=['middle.thing_id = things.id',
                               'things.owner_id = people.id',
                               'people.id in (%s)',
                               'middle.user_id = %s'],
                        params=[idlist, my_id]
                )

Running the code wasn't producing all the correct results. Many were missing. Tracing the code and looking at the returned values, I noticed something: all the results applied to only one user, rather than to the set that I was trying to get. That realization led me to the problem.

My intent was that a SQL statement would get executed that had something like this in the middle:

.. AND people.id in (12,31,62) AND ..

The first piece of unhelpful magic was that the parameters were strings, so the ORM helpfully quoted them for me. So the SQL actually had this:

.. AND people.id in ('12,31,62') AND ..

Then MySQL helpfully converted that string into the needed integer for me, so that the SQL executed as if it were:

.. AND people.id in (12) AND ..

So of course the query was returning results for just one user.

The last strange thing (I'm not sure whether to call it helpful magic because I don't know why anyone would find this helpful) was that the SQL statement tracing provided by the ORM for some reason doesn't display the quoting applied by the ORM. String parameters are displayed without their quote marks. The SQL trace for the original code displayed:

.. AND people.id in (12,31,62) AND ..

which lulled me into believing that the query being executed was the one I wanted.

The fix was a simple one: use Python string replacement instead of ORM parameter replacement to build the query:

idlist = ','.join([str(i) for i in ids])
tlist = things.get_list(
                        tables=['middle','people'],
                        where=['middle.thing_id = things.id',
                               'things.owner_id = people.id',
                               'people.id in (%s)' % idlist,
                               'middle.user_id = %s'],
                        params=[my_id]
                )

This avoided the string quoting, which produced the correct SQL, which produced the correct results.

Note: there may have been a better way to express this. One of the difficulties in doing complex work with an ORM is having to learn two languages: SQL, and the ORM. The simplified syntax provided functionally by the ORM is a good thing when it is slight, but once it gets complicated, I find myself thinking in SQL and back-porting to the ORM.