So that happened... (digg, slashdot, and webfaction)

Tuesday 6 May 2008

Last Thursday, I posted the animated CSS Homer, and it was a big hit. Friday morning, it was popular on Digg (over 3000 diggs). The resulting Digg effect was enough for my hosting provider to shut off my site.

I was a cheapskate when I bought my hosting plan from TotalChoice Hosting, looking only for low cost. Their reaction seemed aggravatingly uninformed. The support guy kept referring to the traffic spike as “an attack”. I tried to explain that it was in fact a success, and that they had failed to help me deal with that success. I could understand needing to protect their widely shared servers, but at least they could speak knowledgeably about the event.

He also called it a DDOS, which it was, but only if it stands for Distributed Desirability Of Stuff.

Further angering me was the fact that my email was unavailable, since they simply shut off my entire account. Also, there was a misconfiguration in the 403 page they were serving, so the traffic logs showed every request resulting in another request for a non-existent 403.shtml page. TotalChoice will be the first to point out that they are not the right service for a high-traffic site, but they should at least be conversant in the language of their newly disappointed customers, and know how to correctly shut off accounts.

Saturday morning, the traffic had subsided and the site was reactivated, and I figured I could spend some time researching options for a new provider. Slicehost seemed good if I wanted to go the VPS route, though sysadmin is not my interest or forte, so I was leery of taking on all the responsibility for the machine, however virtual it was.

WebFaction seemed the best choice of the shared providers, with supported Django, and many Django sites hosted.

I was away for the weekend, so I wasn’t actively working on the problem. My site was up, I could now plan my next move.

At least, until I got slashdotted. Now the site was really shut down, and TotalChoice wasn’t too pleased. The only way back online was with a new provider. WebFaction got the gig, because I don’t need complete control over a machine. A shared account with shell access and supported Django would be great. I looked in their forum for Digg effect issues, and saw intelligent conversation. I had dropped them a line outlining my situation, and they made clear that they had dealt with it before and would work with me if such good fortune arose again, but that they would shut down sites if it was the only way to protect the shared servers. In a way, that last caveat reassured me. If they had made a blanket claim that their servers were Digg-proof, it would have smelled of naive or dishonest admins.

Monday I signed up, switched over my domains nameservers, re-uploaded my site, and I was back online. After getting TotalChoice to reactivate my old site, I transfered the blog comments, and now everything should be back as good as new.

It would have been nice to survive the Digg and Slashdotting. Maybe with WebFaction I will next time. I’ve got a new appreciation for slimming down the server needs of my blog. The avatars in comments are something to think about: the Homer post has 70 comments, meaning each page load also generates 70 image requests. One possibility is to offload the image to another service.

The irony in all this is that although I started with TotalChoice because of how inexpensive they were, I’m not paying much more for the WebFaction account.


I can strongly recommend Amazon's S3 solution for serving things like that. You obviously have to "pay for what you use," but in a normal month it's a dollar or two, but when you get slammed, they can deliver terabytes without trouble. I put all the "assets" on there, so anything except the HTML comes from S3, and the HTML is cached.
I call such a thing a succestrophe.
The blurb from the linked page is choice:

"TotalChoice Hosting understands that you demand more from your web host provider. We know you are tired of empty promises and undelivered services. In fact we built our core business model around these simple facts! Whether you need hosting for your business or your personal site, TotalChoice Hosting will be there for you. That is 24/7 service and support."
So far, I have been impressed by Web Faction. They are a vast improvement over my Django experience at Dreamhost. Wonderful support and a decent control panel.
And I already wondered where the site went. Well, not so surprising, as it was a cool post :)
Congrats on the move. WebFaction is by far the best host I've ever used. A friend of mine also on WebFaction got digg'd once and his site stayed up and responsive the whole time.
Ah, a misconfiguration in the 403 page! I should have thought of that myself.
“Successtrophe”: splendid. My word for the day, thank you Jim!
I can heartily recommend WebFaction. Their staff is knowledgeable and eager to help out. I have several sites hosted with them and they keep getting better all the time with new features and better pricing.
For VPS slicehost is the best I have been on, but with the ability to run whatever you like on webfaction, it looks like a great alternative if you don't truly need full control.
I have the pleasure of interacting with Ned in real life. He dealt with all this, far, far, far better than I would have. I'd have been a purple-faced, vein-bulging, fireball-throwing lunatic. Ned was much closer to serene.

Permit me to throw Ned's former hosting company the double middle finger. I hope they suffer catastrophic DDnC attack (Dollar Denial - no Customers) in the near future.
I guess I was surprisingly calm about the whole thing. There were some apoplectic moments when I was still trying to get TotalChoice to help me out, but once it became clear it was a lost cause, the ball was in my court to make the switch. Unlike the time two years ago when I lost my domain name, I knew this was temporary. The first outage meant all those Digg eyeballs didn't see my blog post, but Digg is a firehose of readers who will never come back again anyway. A spike in page views isn't what I'm after, it's additional subscribers I'm interested in. Losing the slashdot eyeballs was too bad, but I knew there was nothing to do to bring the site up under that load. The best I could do was get up at WebFaction quickly to catch the tail end of that wave.

And yes, succestrophe is an awesome word! If you want to improve the blogosphere, somehow convince Jim Flanagan to bring back!
Someone pointed you out to me at Pycon this year but I missed the opportunity to say I enjoy your blog. I do. Anyway... I host some things at webfaction, too, and they are both friendly and really know their stuff. :-) Good luck with the next successtrophe!

Add a comment:

Ignore this:
Leave this empty:
Name is required. Either email or web are required. Email won't be displayed and I won't spam you. Your web site won't be indexed by search engines.
Don't put anything here:
Leave this empty:
Comment text is Markdown.