Tim Bray writes passionately about speed: On Software Performance. It was inspired by a .NET-specific article on MSDN, but Tim's points apply to any software. An excerpt:

1. Design and code your app, trying hard not to do anything really stupid, and striving for flexibility.
2. If it's fast enough, don't worry any more.
3. If it's slow, get out your profiler and measure things until you understand where the problem is.
4. Fix the problem, which may well require major refactoring, but that's OK because that's probably coming at you pretty soon anyhow with the next batch of requirements. Furthermore, you couldn't have avoided it because nobody is smart enough to predict where the bottlenecks will be in a complex application before it's running.

That last point is right on the money. More than once, I've had this experience: We analyzed the bottlenecks, and identified a piece of code that was holding things up. The code had a comment on it along the lines of, "This should really boost performance".