Happy New Year!
The snow’s not even dry on the rooftops of LA and we here at DreamHost already have a pile of resolutions for, as the cool sports video gamers call it, the 2K7.
In 2007 we do solemnly resolve to:
#1. Never get involved in a land war in Asia.
#2. Never go in against a Sicilian when death is on the line!
#3. Become once again renowned the Web-over as a stable, reliable, robust, high-performance webhost!
As those of you who’ve been playing the home game know, we had some troubles in 2006.
But actually, the root of those troubles began WAY back to June 2005, when the building our data center is in informed everybody they were unable to provide any new UPS and generator-backed power, period.
Moving data centers wasn’t really doable back then, and so for the next year or so we were forced into “low-power mode” .. scrapping our Dedicated Servers option and squeezing every last bit of power efficiency we could from our operations, even at a fair amount of expense.
Somehow, we kept going. And going. And going. And gahhhh, you get it.
And really, our service didn’t suffer for it.
But then, exactly one year ago today, something changed that seemed to affect our reputation for the worse ever since.
We started giving away a lot more disk and bandwidth. Like 8 times. As. Much.
That’s when things went downhill.
Well, not really.
In fact, we had exactly the same amount of problems (actually less, per customer!) we’d had the last eight years, but now finally people could put their finger on a REASON for them!
We were overselling!
Clearly, every problem we had stemmed from the simple fact that we gave away too much disk and bandwidth!
Well, I’ve already covered “overselling” plenty, and ALL the quota increases really did was increase the number of new customers we got!
But still, all through 2006 our rep seemed to slowly decline.
Every time we had a server crash; “Overselling.” A network fubar, “They’re overselling.” A panel bug: “Didn’t your mama ever teach you about overselling?” A power outage? “Oh yeah, sign up for DreamHost if you happen to like a fresh bunch of OVERSELLING!!!”
Of course, the power outages didn’t help. Nor did the weird problem between our two core routers that made our entire network suck eggs for six weeks this summer.
But in a way, those power outages were perhaps a blessing in disguise. A disguise that reminded me of a big mob of angry customers.
Those outages forced us, and our building, to really DO something about the power situation… which as you may recall is the real foundation for any stability problems we’ve had in the last 12 months.
After the power outages this summer, the building started literally BLEEDING data center tenants, figuratively.
This had two effects. Ichi, it forced them to start taking their UPS and generator problems seriously, and as of now they actually seem to have things in order. In fact, believe it or not, just TODAY the building experienced a power outage from DWP… and for the first time ever we were not affected at all!
DATE: January 3, 2007
TO: All Garland Tenants
FROM: Timothy J. Moore
RE: DWP Power Outage Today
This is to advise you that at approximately 11:50am today, the Garland Building received a power outage from the Department of Water and Power. The outage lasted less than one minute and all systems worked according to design.
The Building’s loads were transferred to the Emergency Generator System. Upon stabilization of the DWP service, all ATS Switches transferred back to normal DWP power.
Should you have any questions, please do not hesitate to contact the Office of the Building.
The Happy Office of the Building Team
Oh BOY was I ticked OFF when I saw how they stole our signature signature!
Ni, by bleeding those tenants, a lot of power was freed up for us! And by us I mean you! LITERALLY.
Also, we were now of a size (thanks, ironically, to our generous bandwidth and disk allocations!) that expanding to more than one data center was finally feasible.
So, this fall we expanded to two more facilities, with dark fiber connections between all three.
With all this new power available, we were finally able to spend more on hardware! So we did, and have been, and are, and will be! We’ve put many many fewer users per web server, mysql server, and mail server, added load balancers, beefed up our network equipment, and have added new targets (that we now have the power to attain) for server stability.
In fact, we spent over ONE MILLLLION DOLLARS on hardware in November alone! That’s more than we normally spend in a whole quarter! And in fact, things are quantitatively more stable now across our whole system than they’ve ever been in the past.
But our reputation as an “overselling host” seems to linger!
How can we fix it? Aren’t people just going to notice things are a lot better? And start telling their friends?
Won’t they just believe this blog post?
Probably not. It’s a stumper!
Fortunately, I pulled deep into my master-of-public-relations pouch, and pulled out this gem of wisdom:
People aren’t going to consider us a “stable” host until we offer LESS DISK AND BANDWIDTH!
But…ARGH! More disk and bw => more sign ups => more money => more resources => better service!
What to do?
Fortunately, I have a master-of-marketing pouch too (double-major).. so here’s what we’re doing:
Every day, starting tomorrow, the amount of starting disk and bandwidth we offer new customers (this does not affect existing customers at all!) will drop. You can see the amounts here.
(Don’t worry, once you sign up, your disk and bandwidth allocations will grow weekly just like before!)
And we’ll keep dropping them daily until our precious rep is restored!
(Or it cuts into our sign-ups too much.)
(Whichever comes first.)
(Reputation be damned.)