How DreamHost Builds Its Cloud: Selecting Microprocessors
This is post No. 2 in a series of posts about how DreamHost builds its Cloud products. Written by Luke Odom straight from data center operations.
In this post, we are going to be looking at what processor we are using in the new DreamCompute cluster, and how we picked it!
A processor is one of the most crucial components of a machine. The processor, also known as the CPU, is the metaphorical brain of the computer. It does all the “thinking” for the computer. A CPU can have one or more cores. A core is the part of the processor that does the actual computing. The first thing to consider is the instruction set. This is the language that the processor speaks. The popular instruction sets for use in servers are Sparc, ARM, and x86-64. Processors using the Sparc instruction set are made by Oracle. While Linux can run on them, the processors are designed to run Solaris, an operating system that is also made by Oracle. ARM is an instruction set that has been around for 30 years and had primarily been used in embedded devices. Cell phones, TVs, and tablets are some of the many places you will find ARM processors.
Recently, we’ve been testing some servers with ARM processors. The advantage of these servers is they use very little power and have many low-power cores, which is useful in environments where you have many processes running at the same time. The most common instruction set you will find in a data center is x86-64. This is the 64-bit version of the x86 instruction set that has been in use since 1978. Almost all consumer laptop, desktop, and enterprise servers are made using processors based on x86-64. Because of this, we decided to use a processor based on x86-64.
The server x86-64 processor manufacturers are AMD and Intel. AMD, which we used in our beta cluster, last released a major x86-64 server processor in 2012. Since that release, AMD has been working on Zen, a new architecture. Unfortunately, Zen has not yet been released. Because of this, the AMD processors currently for sale are much higher wattage and slower than others on the market.
In the time since the last major AMD update, Intel has released three new generations of chips, each faster and more power efficient. This makes Intel the best choice right now, so we decided to use them. Every year or so, Intel releases a new “generation” of processors. With each generation, Intel does one of two things. They either change the transistor size, making the processor smaller and more power efficient, or they keep the transistor size the same and focus on adding features.
Within the server line of processors, there are four generations currently being produced. Ivy Bridge, which was released in 2012, is a upgrade of the older Sandy Bridge processors with smaller transistors and is denoted with a v2 at the end of the processor name for most server processors. Haswell, released in 2013, is a refinement of Ivy Bridge and uses the same transistor size. For servers, it is denoted as v3. Broadwell, released in 2014, was a refinement of the Haswell processor with smaller transistors and is denoted with a v4. Finally, SkyLake, released last year, is a refinement of Broadwell at the same transistor size and is denoted with a v5.
Individual product lines within the server category are upgraded at different times. Even though a generation has been released, it may be years before a certain product line begins using that generation, and certain product lines may skip over entire generations. The five current product lines within the server category are Atom, Xeon D, Xeon E3, Xeon E5, and Xeon E7. The Atom line is an ultra-low power line. These aren’t really powerful enough to use in our hypervisors, and the low number of cores would significantly limit the size of virtual machine we could offer. The next line is Xeon D. These are higher wattage and faster than the Atom processors, but still not quite the power we wanted to be able to give our customers.
The next line is the E3. The E3 line is the only server based line that has been upgraded to SkyLake. It has plenty of power, but you are limited to a single processor per system and four cores per processor. The E3 line lacks the density that would make it usable. At the time we were designing the new DreamCompute cluster, the E5 was Haswell-based but we knew Broadwell was coming soon. As we can only use what is currently being produced, we only looked at the Haswell line. The E5 line is marketed to data centers. Within the E5 line, you have both single, dual, and quad processor options with many choices within each of those. The E5 line just might work! The E7 line is last line we looked at. The E7 line was, at the time, Ivy Bridge-based, though we knew Haswell was coming soon. The E7 line is focused on density. E7s have both quad and octo processor options with up to 18 cores per processor. They are primarily used in environments where you need a single computer to be able to do a lot of work. That made E7s a possible, but not ideal, fit for DreamCompute as we probably don’t want that much density.
Now that we knew what processor lines would work, we needed to consider two more factors. We needed to figure how many processors we wanted in each system and how many cores each processor should have. Originally, processors had a single core, but modern x86-64 processors can have up to 22 cores on a single processor. This is especially useful in a shared resource environment (like DreamCompute) where you don’t want the processes of one user to affect everyone else on that hypervisor. Intel also has a technology called “hyperthreading,” where it presents each core to the operating system twice to allow for more efficient use of each core. You can also put multiple processors in each server. You can have up to eight 18-core processors in a system for a total of 144 cores or 288 threads. Though, as we know from our previous cluster, density isn’t everything. We wanted to be able to balance power and density, and limit the single points of failure. Based on the RAM-to-core ratio we wanted and the maximum density we were willing to have, we decided to test dual and quad processor systems with eight to 14 cores in each system. We tested both the E5 dual and quad processor lines as well as the E7 quad processor line.
To pick the processor we wanted to use, we needed to take into consideration a few other factors. But… more on that in another post coming soon!