Grocking Devops

DevOps is the practice of combining metrics and team dynamics for the purpose of enabling continuous change. 

Metrics - business metrics, application metrics, and performance metrics.

Team dynamics - specialists working together closely or in blended roles to affect change in response to metrics.

Continuous change - to the business, product, features, and operational performance.

We have recently switched from a manually configured development environment to a nearly fully automated one using Vagrant, Chef, and a few other tools. With this transition, we’ve moved to an environment where data on the dev boxes is considered disposable and only what’s checked into the SCM is “real”. This is where we’ve always wanted to be, but without the ability to easily rebuild the dev environment from scratch, it’s hard to internalize this behavior pattern. Cruft builds up in the dev boxes, and it starts to slowly drift out of sync unless a lot of manual intervention is applied. Rebuilding the VM is not something you’ll want to do (or have to do) very frequently, but it’s a good feeling to know that you can with very little pain, if it’s needed. This still doesn’t completely eliminate development cruft, but the hope is that it can significantly cut down on it. If you’re worried about running vagrant destroy, that’s a hint that you might have done something wrong.

This post covers our ruby dev environment on Mac hosts. For other platforms it will be slightly different, but the principles are the same - use vagrant to manage your underlying machine configuration, automate as much of the rest as you can, and keep all important data resident off your dev boxes, checked into scm or using another redundant remote mechanism. As someone who’s been doing unix system administration as a sideline to development for about 20 years, I’m really impressed at the extent to which the automation tools have improved in the past few years. I did not have any direct experience with configuring chef when I started this, though I’ve been familiar with the concepts and have used other automated configuration and deployment mechanisms.

The configuration took a few days of solid effort to get up and working to the point where it was usable, and then a few more days to work out the kinks and figure out our preferences. While working through it, it was pretty easy to identify what the next step was given what wasn’t working. Work your way clockwise around the diagram - get vagrant up and running, then do base installs, then work on getting your source tree in place, then get it working, then import dev data.  Instead of distributing ssh keys to the individual VMs, we use agent forwarding to allow each VM to use the keys from its host. The final config consists of only a handful of files. Depending on the complexity of your environment, you may need some additional scripts:

The Vagrantfile which determines the vagrant and chef configuration.

A Cheffile which specifies to librarian-chef which cookbooks to get.

A custom shell script which populates ssh config files, sets up user and database permissions, and performs project checkouts with the appropriate rvm environment.

A shell script to run on the host to add the host’s ssh keys to ssh-agent (probably automated by launchd).

A custom /etc/hosts file to insert into the VM config.

These files are checked into SCM, so bringing up a new dev box is only a few steps (all but the last step only ever need to be done once per machine):

Install virtualbox: https://www.virtualbox.org/wiki/Downloads

Install vagrant: http://downloads.vagrantup.com

Make sure your key is on the server for the dev config checkout.

Check out the dev config repository.

Run the ssh-add script to bring your keys into the ssh-agent.

Install librarian-chef using any local ruby (gem install librarian-chef)

Install the librarian chef vagrant plugin, so vagrant will automatically fetch your cookbooks (vagrant plugin install vagrant-librarian-chef)

Set up any local path location and config variable overrides in  ~/.vagrant.d/Vagrantfile (optional)

Bring up your dev environment with vagrant up.

We use vagrant synced folders to mount selected folders in the host’s Dropbox folder and the separate project tree where checkouts are placed, to share these files between the VM and the host, both for letting developers use their own Mac-native environment to edit files in the source tree and for importing data via dropbox (Rather than checking them into scm, our database test packs are distributed via dropbox, and the provisioning script looks for them there to do the initial import.). Chef handles getting the proper ruby versions installed, and then bundler ensures that the gems needed for each project are present. We have used bundle package and checked in our vendor/cache folder, which greatly reduces the amount of time it takes to install gems.

With this configuration, the VM has its own copies of the databases and its own source trees. The databases and local configuration files (with some critical exceptions that are always overwritten at provisioning time) are wiped out in the case of a vagrant destroy, but otherwise preserved. The source trees are always synced with the host, so they will survive the destruction of the VM, but they can of course be manually wiped if needed.

Some random notes:

There was some complexity in having the provisioning shell scripts running as the vagrant user (instead of the default, which is root), with access to the ssh keys via the ssh-agent forwarding. The best solution I found was to make a separate script file, then call it with sudo -u from the provisioner. All of the configuration that needs to be done as the vagrant user (rvm setup, bundler installs, scm checkouts, environment config etc…) goes into that script.

The default VM runs with very little RAM and only one CPU, which makes compiling (especially new rubies under rvm) extremely slow. You’ll want to bump this up.

There’s a balancing act between what should be configured by chef and what’s easier to just do with shell commands. Most of the simple configuration settings and individual packages are easier to do with the command line.

There seems to be an intermittent issue where a suspended VM will be unable to pick up a DHCP ip address upon resuming, which causes it to hang indefinitely. An interim solution (found in this post) to this seems to be to execute VBoxManage guestcontrol <vagrant machine id> exec “/usr/bin/sudo” —username vagrant —password vagrant —verbose —wait-stdout dhclient from the host.

I find it odd that there’s no way to specify local Vagrantfile changes that take precedence over the project files. This means you either have to design your Vagrant config in such a way that you’re setting variables which can be overridden locally, or people have to edit the project Vagrantfile directly (and risk accidentally checking in their customizations). This seems likely to cause problems as different developers are going to want to use machines with different amounts of memory. We’ve temporarily worked around this by using global config variables and only setting them if they’re not already set.

Vagrant uses its own config file for controlling chef commands in a way that differs from chef’s own syntax, meaning that chef configs can’t be reused outside vagrant. I haven’t yet found if there’s a way to import generic reusable chef configs directly.

Our Vagrantfile (with some custom config removed) looks like this:

Bryan Berry:

Writing reams of documentation sucks. Chef drastically reduces the amount of documentation you have to write.

Bash doesn’t scale. Seriously.

Technical Awesomeness

Chef grows with you

You can stop reinventing the wheel

There’s no large scale without automation. Just ask Netflix about their army of monkeys.

Summary