A Storage Story

A whirlwind tour through the various storage technologies used by DreamHost over the years. These days we’re hoping to change how people think about storage by introducing them to Ceph. Ceph is a distributed object store and filesystem designed to provide excellent performance, reliability, and scalability. This talk will describe the Ceph architecture and the impact it has on systems operations. The DreamObjects Ceph cluster weighs in at over 2 petabytes of raw storage capacity while the initial DreamCompute cluster boasts over 3. Chef has helped DreamHost automate the configuration and management of large Linux based storage clusters atop commodity hardware. Ceph and Chef allow us to manage these clusters with two engineers who also share bandwidth with other projects.