#buildduty

I've just finished my milk

I wanted to post a follow-up to plug an accidental omission in catlee's 2016 RelEng Retrospective. Buildduty is one of those things we rarely talk about publicly; it is simultaneously on the front-lines of developer support for releng, and yet somehow consistently flies under the radar.

2016 was the first year where we had full-time buildduty support from Softvision. Previous to their involvement, buildduty was a responsibility that rotated through the rest of releng. We tried various different models and cadences, but because of the interrupt-driven nature of the work, buildduty was avoided like a pariah or an Australian table wine.

As we approach 2017, I now consider buildduty a solved problem. I can't imagine what releng life would be like without the buildduty help we get from Alin and Andrei. Yes, there's still lots of work to do, but I no longer worry about *how* we'll get it done.

This is largely a testament to the work put in by Softvision. In fact, in the spring of 2016, there was a personnel change with one of our buildduty contractors leaving for another opportunity. Softvision trained up a replacement (Andrei) in the background and subbed them in, and they barely missed a beat.

I'd like to also explicitly call out the increasing amount of buildduty management help provided by kmoir.

How exactly is buildduty helping? The list is long, varied, and growing: machine loans, package uploads, puppet environment changes, config changes, platform capacity work, debugging help, etc.

Here are some aggregate numbers from 2016:

Bugs resolved: 671

Machines loaned: 106

Patches submitted: 59

That last number is the one that intrigues me the most: 59 patches submitted means that buildduty has moved past simple maintenance and are now proactively fixing problems for releng. Indeed, over the past year buildduty has taken on an increasing ownership role of our soon-to-be legacy buildbot infrastructure. We now routinely ask them to make changes to the buildbot configs on their own...with proper review, of course!

Previously

Buildduty is the Mozilla release engineering (releng) equivalent of front-line support. It's made up of a multitude of small tasks, none of which on their own are particulary complex or demanding, but taken in aggregate can amount to a lot of work.

It's also non-deterministic. One of the most important buildduty tasks is acting as information brokers during tree closures and outages, making sure sheriffs, developers, and IT staff have the information they need. When outages happen, they supercede all other work. You may have planned to get through the backlog of buildduty tasks today, but congratulations, now you're dealing with a network outage instead.

Releng has struggled to find a sustainable model for staffing buildduty. The struggle has been two-fold: finding engineers to do the work, and finding a duration for a buildduty rotation that doesn't keep the engineer out of their regular workflow for too long.

I'm a firm believer that engineers *need* to be exposed to the consequences of the software they write and the systems they design:

DevOpsSupport - You write the software, you get a shift carrying the pager for it, and you get a shift answering customer calls about it.

— mhoye (@mhoye) July 15, 2015

I also believe that it's a valuable skill to be able to design a system and document it sufficiently so that it can be handed off to someone else to maintain.

Starting this week, we're trying something new. We're shifting at least part of the burden to management: I am now managing a pair of contractors who will be responsible for buildduty for the rest of 2015.

Alin and Vlad are our new contractors, and are both based in Romania. Their offset from Mozilla Standard Time (aka PST) will allow them to tackle the asynchronous activities of buildduty, namely slave loans, non-urgent developer requests, and maintaining the health of the machine pools.

It will take them a few weeks to find their feet since they are unfamiliar with any of the systems. You can find them on IRC in the usual places (#releng and #buildduty). Their IRC nicks are aselagea and vladC. Hopefully they will both be comfortable enough to append |buildduty to those nicks soon. :)