DreamHost Announcements

Yoohoo! What the f— just happened to my database?

We had a little foobar with our registrar database tonight.

We had a little foobar with our registrar database tonight. Unbelievably it wasn’t my fault, although in a way everything is, since it’s my fooing database!

A number of statuses were wiped out, which resulted in about 4500 customers getting an email saying their domain had been successfully transferred to us! Yay!

Unfortunately, the last good backup was two days old.

How did we fix it?

First, Sage called me and told me all this stuff he’d like me to do.
Then, Sage mostly just did it all anyway.

Sage turned off the registrar service so no more emails would go out until everything was fixed.

Then, I figured out who just got emailed the crazy email, and sent them all another email saying “PLEASE IGNORE”. This stemmed the growing flood of confused support requests.

Andrea (a.k.a. the CULPRIT) also went in and replied to everyobdy who’d already written in.

Sage went in and wrote a script to quickly update the statuses in our system from the appropriate registries.

This didn’t help for domains with statuses like “pending-registration” and “pending-transfer”. For those we looked at a second table in our database with related data.

In fact, we decided to make the most of this opportunity to change the way all our code gets statuses from the database.. so now it uses that second table for all the special statuses we don’t get from the registry itself.. something Sage had said he’d wanted to do anyway.

Just not necessarily at 1am on a school night.

Once we’d made all the code changes necessary we copied them live and tried running our registrar service again. It pretty much worked, so we turned it back on for good and hit the hay!

Except for me, who decided to post about it here.


About the author

Josh Jones