#facebook engineering

What Makes Facebook Developers Productive

Google, Amazon and Facebook all are using very aggressive Continuous Delivery workflows and have been doing so for years.

Weekly, daily or even hourly production deployments are a commonly-practiced, enterprise-scale software development life cycle.

Today, numerous "Web-scale" organizations (notably Google, Amazon and Facebook) practice continuous deployment, with real-time production monitoring and a lot of exploratory testing taking place in production as well.

Here are a dozen articles about Web-scale continuous deployment in the enterprise

After reading these links it should be clear that Continuous Delivery is Yet Another Mainstream Software Methodology, just like Agile and Waterfall. Please reach out to me on Twitter if if you think you can still make the case -- in light of this evidence -- that Continuous Delivery is still "untried" or especially risky ;-)

One other thing to take note of here: almost all these articles are old. Some were written 2 years ago or more. Rapid iterative development may or may not work for a specific organization -- but the general problem of implementing Continuous Delivery is clearly well-solved and has been so for a while now.

Google practices Continuous Delivery

At Google, 15,000 engineers work from the HEAD revision of a single Perforce trunk.

50% of the code will be changed in any given month.

Google's test infrastructure is legendary and they've written a comprehensive book about how they perform QA while continuously releasing.

They've also put a lot of effort into scaling Perforce.

Here is a fantastic deep-dive into Google's deployment pipeline:

Amazon practices Continuous Delivery

At Amazon, new code is deployed to production at a staggering rate of once every 11.6 seconds during a normal business day.

That's 3,000 production deployments per day.

They've invested an enormous amount of time and money into creating an architecture that facilitates small, orthogonal, frequent code pushes.

Here's Amazon's Jon Jenkins breaking down their deploy stats at the O'Reilly Velocity conference:

Facebook practices Continuous Delivery

At Facebook, each of 5,000 engineers commits to trunk HEAD at least once a day and the code at trunk HEAD is pushed to production once daily.

Facebook has no dedicated QA team. All responsibility for testing rests with the software engineers.

They've invested heavily in infrastructure that provides zero-downtime deployment at Facebook scale.

Here's Facebook's release manager, Chuck Rossi, going into detail about how Facebook engineers balance their experiments against the risk of breaking some fundamentally important part of the site:

(function(d, s, id) { var js, fjs = d.getElementsByTagName(s)[0]; if (d.getElementById(id)) return; js = d.createElement(s); js.id = id; js.src = "//connect.facebook.net/en_US/all.js#xfbml=1"; fjs.parentNode.insertBefore(js, fjs); }(document, 'script', 'facebook-jssdk'));

Post by Facebook Engineering.

UPDATED January 21, 2014

I meant the adjective "mainstream" in the sense of "not dangerous." For example: "The Ramones are so mainstream, I only listen to Norwegian Death Metal." Based on the large amount of feedback I have received, this is not everyone's default definition of "mainstream." Hopefully adding this paragraph to the post will clear up such ambiguity for future readers.

Once again, I did not mean to imply that everyone is doing CD. Everyone is not doing CD! But, CD is no longer the risky experiment it was in 2010 when Chad Dickerson hired me to help scale the CI system at Etsy (which is how I got involved in this whole discussion in the first place). Today CD is a mature option and I think it is the best option available. But there are certainly other ways to build software and lots of people use those other methodologies to great success.

High School is where I fell in love with programming, was always fascinated by how one can use code lines to create. Observing my high school teacher creating a simple HTML offline site ignited the fire.