Behind the Scenes of a botched Ruby on Rails upgrade

Rails Logo As many of you noticed last night and this morning, our attempted upgrade of Ruby on Rails from 1.0 to 1.1 did not go as well as we had hoped. We have now reverted back to the stable 1.0 release while we spend some time testing. A few different factors had a role in causing this mishap including the ever present ‘human error’.

The Ruby on Rails software is really 6 or so separate pieces of software that all must have the correct versions installed. If one of the pieces is not properly installed the whole thing breaks. That in itself is not much of a problem if you use the recommended ‘gem’ tool to upgrade the software, handling any software dependencies for you along the way. A lot of Ruby software is distributed in these little packages called rubygems (clever, eh?) and the gem tool is used to manage them. The gem and rubygem system is somewhat like an improved version of Perl’s CPAN or PHP’s PEAR, if you are familiar with those. This is all well and good but we don’t rely on the gem and Rubygem system to handle our Ruby software upgrades for us. I’ll step back in time a little to explain why.

Several years back as our fleet of servers started to grow to be somewhat large we realized we could not continue to manage each server independently. To deal with the situation we developed methods and procedures to manage our servers as groups keeping the servers in a group all configured and set up as identically to one another as possible. Debian’s excellent apt-get software package management system was the basis for the software installation side of things, and still is today. We rarely distribute any software to our hosting servers without putting it into a Debian package and anything that will be installed on more than one server must be packaged up first. That includes all perl modules, php software, and (you guessed it!) rubygems.

As things are now those 6 or so individual pieces of software that form Ruby on Rails are all individual Debian packages and they must all be upgraded simultaneously, preserving the proper dependencies. We have coded the dependencies into the Debian packages themselves to make it a little easier, but they are still separate packages that must be upgraded and built separately. In hindsight it may not have been the best decision to put them into separate packages and that is something we will likely change now! So, the human error part came into play when two of the upgraded packages were not built correctly. Human error is the component of any technical problem that’s simultaneously the easiest and hardest to correct. You don’t have to change anything to correct it but you can also never really be sure it won’t happen again. In this case it will take plenty of time to get the egg off of our faces so we’re pretty sure that will cement things in our heads plenty.

Unfortunately, even if we had done a perfect job upgrading the Ruby on Rails software, a lot of websites still would have broken. It seems as if the Rails developers’ claims of an easy upgrade were perhaps a little over-enthusiastic. However, we still can’t just wait around until every Rails user we have updates their code. We have plenty of customers who would rather we do upgrade Rails as soon as possible after every new release even if it might break their website just so they can have access to that ever tempting ‘bleeding edge’. Ruby on Rails is still in a very rapid development cycle with frequent updates with hundreds of changes and it is somewhat likely that the next few upgrades may cause some small problems as well.

Fortunately, there is an easy solution!

As of a couple of versions ago, Ruby on Rails added the very nifty ability to freeze your Rails application at the version of the Ruby on Rails software currently installed system-wide. Once that’s done any changes to the software on the server (ie, changes done by us!) will not affect your application. This is highly recommended for any applications running on a shared hosting platform or any other setup where you do not have control over the Ruby on Rails installation.

Here’s how…
From the application directory, run rake freeze_gems. That will copy all of the Ruby on Rails software (that does not include other rubygems!) into a local area. It must be done for every Rails application independently. When you want to switch back and use the version of the software installed system-wide you just run rake unfreeze_gems.

So, everyone go freeze your applications now!