Power Outage Update

Evacuation!

Okay, this sucked. Literally 10 seconds after Nate posted that last entry, we received an announcement over our loud speakers that our building was being evacuated and we had to leave immediately! (Also, sorry about that post being temporarily taken down, Nate thought it would be better to remove it since it was so dumb, but methinks better to just add something else!)

Unfortunately, shortly thereafter we figured out (thanks to EVDO!) our sites were no longer accessible, as our three network providers Level 3, Global Crossing, and Mzima were all down. Our physical servers were okay, and probably really happy sitting in the data center (we were no longer able to access) with nothing to do.

But that didn’t last long. Shortly thereafter the entire building where our data center is located’s back-up generators (there are SUPPOSED to be four) stopped working, and all power was gone. We were able to get back into our data center then, and it was like the day after tomorrow or something. Really creepy just walking through rows of dark, quiet, dead server after dark, quiet, dead server. We’ll post some pics from that soon..

So, now, finally power is back up and servers are being restored. We lost a net app and had to replace the head unit from a hot spare. A few servers were of course wonky, and right now three bad cabinet switches are still being worked on (so many web sites ARE still down).

There will be more posts soon, and hopefully everything will be up within the next hour.

Also, just FYI for people who don’t know, we keep http://status.dreamhost.com/ in San Jose, completely off our network, just to update people in times like this when our panel (still down) and everything is caput. This is actually the first time we’ve used it!

Now to go update it…