Holy crap guys and girls.
"NightmareHost" is a term that's been thrown around by upset customers since the very beginning. Pick a name like DreamHost and you're pretty much asking for it.
Sticks and stones, right? Not this time. This is the first month where we feel like we've actually earned the title. And we're just as... nonplussed about it as you are.
January was a perfect storm of software deployment issues, critical networking hardware failures, and a security situation that prompted a large-scale password reset.
If you'd like to relive the magic of the last 30 days, January's troubles were documented in great detail as-they-happened over on our system status blog:
This should not come as much of a surprise - even if you didn't notice any problems with your site in January, the email we sent out to all our customers was hard to miss.
We let you down - of that we are acutely aware. All I can say is we're sorry beyond words and we're working on fixing things - a lot of things. We're hopeful that once we come out on the other side we'll again be that gleaming paragon of hosting excellence that you were supposed to be thinking of us as all along.
There's really nothing funny about January. Sorry.
The good news is we've got lots to announce this month, so let's get this show on the road.
Simon Says... (Written by CEO Simon Anderson!)
As Brett has said, January was not a good month for our level of service. And on behalf of all of us here at DreamHost, I want to apologize and let you know what the issues were, and what we're doing to fix them.
First up, security. We're now a mid-sized global web hosting company with over 300,000 customers and 1.2 million domains under management. That means we get our fair share of hack attempts from around the world. Sometimes it's just kids looking to prove themselves, other times it's more malicious than that - people wanting to take down customer blogs or websites because they don't agree with the opinions expressed on them; people wanting to steal other folks domains, that kind of thing. We put a lot of people and dollars into our security software and systems and protect our customers from these attempts every day. But in January a hacker got through our defenses and we took the rapid action of resetting all customers' FTP/shell passwords to prevent any malicious activity. In the process we identified a whole slate of improvements to our security systems, software and data management that we've already been implementing. Security is one of those issues that we can't give too much detail on, but rest assured we're on it and have upped our game substantially so that this kind of event is prevented in future. And as outlined in the newsletter below, we're also asking you to participate by using strong passwords and other best practices.
The second issue in January was related to our network. Given our unlimited bandwidth policy for shared, VPS, and dedicated customers, we have a network that has been designed for speed using tried and tested network equipment. However, the complexity of our routing and switching systems has increased as our global customer base has grown. As a result, we experienced some unplanned network downtime in January due to network configuration errors, plus the testing of some new distributed denial of service (DDoS) equipment. We've already been in the process of a focused effort since late last year to upgrade and streamline our network for the next 5 years of customer needs. But these issues have accelerated our planned efforts in this area. We'll be investing over 1 million dollars in network upgrades in 2012, including a new US east coast data center to better serve our European and east coast customers.
Third, we had an issue with software updates in January that resulted in some service downtime. The issue arose because of the way we have approached security updates for the operating system that runs on most of our servers. We've typically tested security updates to ensure they install correctly, but our testing of whether they break anything has typically been "light" due to the fact that we haven't seen any such issues in the past. Well this has proved to be a flawed approach, because one such update in January went badly wrong, and automatically removed some critical software on production servers, causing downtime and an understandable amount of customer angst. We've immediately changed our software update process to fully test all software going into production, and we'll be acutely focused going on ensuring that any necessary software upgrades are performed efficiently and quickly in planned maintenance windows going forward, with minimal downtime.
Finally, a quick word about our team. We have very dedicated, smart and super friendly people on our support, systems administration, development, engineering and data center operations teams. They work long hours and are absolutely passionate about delivering you an awesome and flexible web hosting service at a great price. When there are service issues like we've seen in the last month, they pull out all the stops – nights, weekends, 24/7 – to fix the issues and plan out changes to give you a better experience for you in future.
I'm fully committed to ensuring we deliver a reliable, flexible and useful service for you going forward. And I believe we're adapting and focusing on the right priorities and investments to continue to deliver a great hosting service and experience.
(Now back to Brett!)
Web Panel Password Improvements
The recent focus on passwords has meant that we've been pretty focused on passwords...recently.
It occurred to us that we weren't doing all that we could to help you select strong, uncrackable passwords for access to your data.
Over the next few days you'll start to see "password strength" meters appear anywhere and everywhere that passwords can be set throughout your account control panel.
Try to pick something dumb like "cat" or "password" and the meter will glow red. Red is an indicator of horrible passwordery - green's where you want to be.
"catdogpets123" will probably break you into yellow.
"c@td0gp3ts123" will get you in the green for sure.
"Cat password? 123? Haha cats are crazy!" will get you green and put you on the honor roll.
So why go to all this trouble?
If someone were to gain access to your password (which we do store in an encrypted form), it's several orders of magnitude easier and faster for them to guess it if it's something short and simple.
The more complex and lengthy your password is, the better off you are. Make them good enough and your passwords will become totally un-brute-forceable by even the most powerful computers on the planet.
Some password management is out of our hands, however. If you've got a WordPress installation sitting on your DreamHost account, for example, you might want to consider if the password you're using within WordPress is really the best option.
We would strongly encourage you to consider changing your web panel password soon to ensure you're in the green. You'll never know if you don't try!
Honestly, just watching the colors change is enough for me. I've changed my own password two dozen times in the last 24 hours and it never gets old.
WordPress - Now Leaner and Meaner
Many of you have made it clear to us that we've been a little TOO helpful in getting WordPress up and running on your account.
We've tried to give you so much more than the standard, out-of-the-box WordPress experience because we wanted you to understand just how powerful and pretty it could be.
To that end we've had our One-Click WordPress installer throw in a bunch of themes from Automattic.com (WordPress's online parent) as well as some standard plugins to get you up and running with as little legwork required on your part as possible.
Turns out a lot of you just ended up deleting all of those helpful add-ons and grew quite frustrated in the process. That makes total sense because some people prefer cookie dough ice cream with sprinkles, and some people ONLY WANT VANILLA, DAMNIT.
We've now made those themes and plugins entirely optional.
The next time you use our One-Click Installer to install WordPress on your DreamHost account, simply un-check the new "Deluxe Install" checkbox and you'll get the standard post-installation WordPress experience that the rest of the Internet has become accustomed to.
You talked, and we listened! Less is more!
We're Coming To Your Town...Maybe!
We'll be hitting the road over the next few months in what we're calling the "DreamHost Reach Out And Touch You Tour - 2012!"
We've always welcomed your feedback over email, but now you'll have an opportunity to let us know what's workin' for ya - and what's not - to our faces!
For three glorious hours you'll be able to hang out and share adult beverages, any-age appetizers, and anything else you've wanted to get off your chest with key members of the DreamHost team.
We plan to touch hundreds of people in the following six cities:
Los Angeles - March 1st, 2012
(These dates are subject to change.)
Why those six? Because that's where most of you happen to live!
We expect these free events to fill up quickly, so be sure to register yourself soon:
Tickets are now available for Los Angeles, but keep an eye on our Twitter and Facebook feeds to learn when tickets to other cities will become available.
All attendees will receive a nice little memento of the evening, but the emotional scars should last a lifetime.
DreamHost WTF: DreamHost Donations
What's That Feature? It's DreamHost Donations!
Web hosting doesn't grow on trees. At the end of the day there's a bill to pay. A modest bill (we hope!) but a bill nonetheless.
We've devised a way for you to earn completely free hosting on the merits of your site's content and quality alone!
DreamHost Donations is a way for you to solicit donations from your websites' visitors that will be applied directly to your DreamHost bill - and only to your hosting bill.
You can't cash these donations out - they are immediately applied toward any costs that you incur (or will incur) with DreamHost.
All you've got to do is place one simple link somewhere - anywhere - on your website. We'll give you the link and a button - the rest is up to you!
Then just sit back, keep cranking out content that the Internet can't get enough of, and watch all that untouchable cash roll in.
The DreamHost Site Of The Month is a coveted title reserved for a select few DreamHost customers. Past winners have had their lives irreversibly changed - mostly for the better.
As voted by you, the winners from JANUARY are as follows...
Great photos - so crisp! Like a ripe apple.
I'd say the experiment was a success!
Remember, you can always click through to the DreamHost Site Of The Month section of your account control panel at any time to submit your site, vote on candidates, and view all winners past and present. (Winners appear in bold.)
Nice work, everyone!
See ya next month!
See ya next month!
P.S. - We apologize if you smiled reading any of this. If smiles did happen, feel free to unsubscribe.