#releng

You may recall two short months ago when we moved Linux and Android nightlies from buildbot to TaskCluster. Due to the train model, this put us (release engineering) on a clock: either we'd be ready to release a beta version of Firefox 53 for Linux and Android using release promotion in TaskCluster, or we'd need to hold back our work for at least the next cycle, causing uplift headaches galore.

I'm happy to report that we were able to successfully release Firefox 53.0b1 for Linux and Android from TaskCluster last week. This is impressive for 3 reasons:

Mac and Windows builds were still promoted from buildbot, so we were able to seamlessly integrate the artifacts of two different continuous integration (CI) platforms.

The process whereby nightly builds are generated has always been different from how we generate release builds. Firefox 53.0b1 represents the first time a beta build was generated using the same taskgraph we use for a nightly, thereby reducing the delta between CI builds and release builds. More work to be done here, for sure.

Nobody noticed. With all the changes under the hood, this may be the most impressive achievement of all.

A round of thanks to Aki, Johan, Kim, and Mihai who worked hard to get the pieces in place for Android, and a special shout-out to Rail who handled the Linux beta while also dealing with the uplift requirements for ESR52. Of course, thanks to everyone else who has helped with the migration thus far. All of that foundational work is starting to pay off.

It's been a while. How are you?

Modernize infrastructure:

We finally closed the 9-year-old bug requesting that we redirect all HTTP traffic to hg.mozilla.org to HTTPS! Many thanks to everyone who helped ensure that automation and other tools continued to work normally. Not every day you get to close bugs that are older than my kids. https://bugzilla.mozilla.org/show_bug.cgi?id=450645

The new TreeStatus page (https://mozilla-releng.net/treestatus) was finally released by garbas with a proxy in place of old url.

Improve Release Pipeline:

Initial work on Uplift dashboard has been done by bastien and released to production by garbas. https://shipit.mozilla-releng.net/release-dashboard

Releng had a workweek in Toronto to plan how release promotion will work in a TaskCluster world. With the uplift for Firefox 52 rapidly approaching (see Release below), we came up with a multi-phase plan that should allow us to release the Linux and Android versions of Firefox 52 from TaskCluster, with the Mac and Windows versions still being created by buildbot.

Improve CI Pipeline:

Alin and Sebastian disabled Windows 10 tests on our CI. Windows 10 tests will be reappearing later this year once we move datacentres and acquire new hardware to support them. https://bugzilla.mozilla.org/show_bug.cgi?id=1330999

Andrei and Relops converted some Windows talos machines to run Linux64 to reduce wait times on this platform. https://bugzilla.mozilla.org/show_bug.cgi?id=1337452

There are some upcoming deadlines involving datacentre moves that, while not currently looming, are definitely focusing our efforts in the TaskCluster migration. As part of the aforementioned workweek, we targeted the next platform that needs to migrate, Mac OS X. We are currently breaking out the packaging and signing steps for Mac so that they can be done on Linux. That work can then be re-used for l10n repacks *and* release promotion.

Operational:

Since most of our Linux64 builds and tests have migrated to TaskCluster, Alin was able to shut down many of our Linux buildbot masters. This will reduce our monthly AWS bill and the complexity of our operational environment. https://bugzilla.mozilla.org/show_bug.cgi?id=1335435

Hal ran our first “hard close” Tree Closing Window (TCW) in quite a while on Saturday, February 11 (https://bugzilla.mozilla.org/show_bug.cgi?id=1324148). It ran about an hour longer than planned due to some strange interactions deep in the back end, which is why it was a "hard close." The issue may be related to occasional "database glitches" we have seen in the past. This time IT got some data, and have raised a case with our load balancer vendor.

Release:

We are deep in the beta cycle for Firefox 52, with beta 8 coming out this week. Firefox 52 is an important milestone release because it signals the start of another ESR cycle.

Every year, hundreds of release engineers swim upstream because they're built that way.

Last week, we (Mozilla release engineering) had a workweek in Toronto to jumpstart progress on the TaskCluster (TC) migration. After the success of our previous workweek for release promotion, we were anxious to try the same format once again and see if we could realize any improvements.

Prior preparation prevents panic

We followed all of the recommendations in the Logistics section of Jordan's post to great success.

Keeping developers fed & watered is an integral part of any workweek. If you ever want to burn a lot of karma, try building consensus between 10+ hungry software developers about where to eat tonight, and then finding a venue that will accommodate you all. Never again; plan that shit in advance. Another upshot of advance planning is that you can also often go to nicer places that cost the same or less. Someone on your team is a (closet) foodie, or is at least a local. If it's not you, ask that person to help you with the planning.

What stage are you at?

The workweek in Vancouver benefitted from two things:

A week of planning at the All-Hands in Orlando the month before; and,

Rail flying out to Vancouver a week early to organize much of the work to be done.

For this workweek, it turned out we were still at the planning stage, but that's totally fine! Never underestimate the power of getting people on the same page. Yes, we did do *some* hacking during the week. Frankly, I think it's easier to do the hacking bit remotely, but nothing beats a bunch of engineers in a room in front of a whiteboard for planning purposes. As a very distributed team, we rarely have that luxury.

Go with it

...which brings me to my final observation. Because we are a very distributed team, opportunities to collaborate in person are infrequent at best. When you do manage to get a bunch of people together in the same room, you really do need to go with discussions and digressions as they develop.

This is not to say that you shouldn't facilitate those discussions, timeboxing them as necessary. If I have one nit to pick with Jordan's post it's that the "Operations" role would be better described as a facilitator. As a people manager for many years now, this is second-nature to me, but having someone who understands the problem space enough to know "when to say when" and keep people on track is key to getting the most out of your time together.

As catlee has already mentioned, yesterday we shipped the first nightly builds for Linux and Android off our next-gen Mozilla continuous integration (CI) system known as TaskCluster. I eventually want to talk more about why this important and how we got to here, but for now I'd like to highlight some of the people who made this possible.

Thanks to Aki's meticulous work planning and executing on a new chain of trust (CoT) model, the nightly builds we now ship on TaskCluster are arguably more secure than our betas and releases. Don't worry though, we're hard at work porting the chain of trust to our release pipeline. Jordan and Mihai tag-teamed the work to get the chain-of-trust-enabled workers doing important things like serving updates and putting binaries in the proper spots. Kim did the lion's share of the work getting our task graphs sorted to tie together the disparate pieces. Callek wrangled all of the l10n bits. On the testing side, gbrown did some heroic work getting reliable test images setup for our Linux platforms. Finally, I'd be remiss if I didn't also call out Dustin who kept us all on track with his migration tracker and who provided a great deal of general TaskCluster platform support.

I've just finished my milk

I wanted to post a follow-up to plug an accidental omission in catlee's 2016 RelEng Retrospective. Buildduty is one of those things we rarely talk about publicly; it is simultaneously on the front-lines of developer support for releng, and yet somehow consistently flies under the radar.

2016 was the first year where we had full-time buildduty support from Softvision. Previous to their involvement, buildduty was a responsibility that rotated through the rest of releng. We tried various different models and cadences, but because of the interrupt-driven nature of the work, buildduty was avoided like a pariah or an Australian table wine.

As we approach 2017, I now consider buildduty a solved problem. I can't imagine what releng life would be like without the buildduty help we get from Alin and Andrei. Yes, there's still lots of work to do, but I no longer worry about *how* we'll get it done.

This is largely a testament to the work put in by Softvision. In fact, in the spring of 2016, there was a personnel change with one of our buildduty contractors leaving for another opportunity. Softvision trained up a replacement (Andrei) in the background and subbed them in, and they barely missed a beat.

I'd like to also explicitly call out the increasing amount of buildduty management help provided by kmoir.

How exactly is buildduty helping? The list is long, varied, and growing: machine loans, package uploads, puppet environment changes, config changes, platform capacity work, debugging help, etc.

Here are some aggregate numbers from 2016:

Bugs resolved: 671

Machines loaned: 106

Patches submitted: 59

That last number is the one that intrigues me the most: 59 patches submitted means that buildduty has moved past simple maintenance and are now proactively fixing problems for releng. Indeed, over the past year buildduty has taken on an increasing ownership role of our soon-to-be legacy buildbot infrastructure. We now routinely ask them to make changes to the buildbot configs on their own...with proper review, of course!

Welcome back. As the podiatrist said, lots of exciting stuff is afoot.

Modernize infrastructure:

The big news from the past few weeks comes from the TaskCluster migration project where we now have nightly updates being served for both Linux and Android builds on the Date project branch. If you’re following along in treeherder, this is the equivalent of “tier 2” status. We’re currently working on polish bugs and a whole bunch of verification work before we attempt to elevate these new nightly builds to tier 1 status on the mozilla-central branch, effectively supplanting the buildbot-generated variants. We hope to achieve that goal before the end of 2017. Even tier 2 is a huge milestone here, so cheers to everyone on the team who has helped make this happen, chiefly Aki, Callek, Kim, Jordan, and Mihai.

A special shout-out to Dustin who helped organize the above migration work over the past few months but writing a custom dependency tracking tool. The code is here https://github.com/taskcluster/migration and you can see output here: http://migration.taskcluster.net/ It’s been super helpful!

Improve Release Pipeline:

Many improvements to Balrog were put into production this past week, including one from a new volunteer. Ben blogged about them in detail.

Aki released several scriptworker releases to stabilize polling and gpg homedir creation. scriptworker 1.0.0b1 enables chain of trust verification.

Aki added multi-signing-format capability to scriptworker and signingscript; this is live on the Date project branch.

Aki added a shared scriptworker puppet module, making it easier to add new instance types. https://bugzilla.mozilla.org/show_bug.cgi?id=1309293

Aki released dephash 0.3.0 with pip>=9.0.0 and hashin>=0.7.0 support.

Improve CI Pipeline:

Nick optimized our requests for AWS spot pricing, shaving several minutes off the runtime of the script which launches new instances in response to pending buildbot jobs.

Kim disabled Windows XP tests on trunk, and winxp talos on all branches (https://bugzilla.mozilla.org/show_bug.cgi?id=1310836 and https://bugzilla.mozilla.org/show_bug.cgi?id=1317716) Now Alin is rebalancing the Windows 8 pools so we can enable e10s testing on Windows 8 with the re-imaged XP machines. Recall that Windows XP is moving to the ESR branch with Firefox 52 which is currently on the Aurora/Developer Edition release branch.

Kim enabled Android x86 nightly builds on the Date project branch: https://bugzilla.mozilla.org/show_bug.cgi?id=1319546

Kim enabled SETA on the graphics projects branch to reduce wait times for test machines: https://bugzilla.mozilla.org/show_bug.cgi?id=1319490

Operational:

Rok has deployed the first service based on the new releng microservices architecture. You can find the new version of TryChooser here: https://mozilla-releng.net/trychooser/ More information about the services and framework itself can be found here: https://docs.mozilla-releng.net/

Release:

Firefox 50 has been released. We’re are currently in the beta cycle for Firefox 51, which will be extra long to avoid trying to push out a major version release during the busy holiday season. We are still on-deck to release a minor security release during this period. Everyone involved in the process applauds this decision.

OK, I've given up the charade that these are weekly now. Welcome back.

Modernize infrastructure:

Thanks to an amazing effort from Ed Morley and the rest of the Treeherder team, Treeherder has been migrated to Heroku, giving us significantly more flexible infrastructure.

Git-internal is no longer a standalone single point of failure (SPOF)! A warm standby host is running, and repository mirroring is in place. We now also have a fully matching staging environment for testing.

Improve Release Pipeline:

Aki and Catlee attended the security offsite and came away with todo items and a list to prioritize to improve release security.

Aki released scriptworker 0.8.0; this gives us signed chain of trust artifacts from scriptworkers, and gpg key management for chain of trust verification.

Improve CI Pipeline:

We now have nightly Linux64, Linux32 and Android 4.0 API15+ builds running on the date branch on taskcluster. Kim’s work to refactor the nightly task graph to transform the existing build “kind” into a signing “kind” has made adding new platforms quite straightforward. See https://bugzilla.mozilla.org/show_bug.cgi?id=1267426 and https://bugzilla.mozilla.org/show_bug.cgi?id=1277579 for more details.

There is still some remaining setup to be done, mostly around updates and moving artifacts into the proper locations (beetmover). Releng will then begin internal testing of these new nightlies (essentially dogfooding) to ensure that important things like updates are working correctly before we uplift this code to mozilla-central.

We hope to make that switch for Linux/Android nightlies within the next month, with Mac and Windows coming later this quarter.

Operational:

During a recent tree closing window (TCW), the database team managed to successfully switch the buildbot database from MyISAM to InnoDB format for improved stability. This is something we’ve wanted to do for many years and it’s good to see it finally done.

Release:

We’re currently on beta 10 for Firefox 50. This is noteworthy because in the next release cycle, Firefox 52 will be uplifted to the Aurora (developer edition) and Firefox 52 will be the last version of Firefox to support Windows XP, Windows Vista, and Universal binaries on Mac. Firefox 52 is due for release in March of 2017. Don’t worry though, all these platforms will be moving to the Firefox 52 ESR branch where they will continue to receive security updates for another year beyond that.