Given the lost productivity of build times how are game dev companies trying to speed them up? Is it just a matter of shoving as much RAM and CPU cores into the workstations as possible or do things like cloud synching issues put a hard cap on how much throwing money at the problem helps?
The larger the team gets, the more effort and resources the studio spends on reducing build times. This is because it's a multiplicative effect - reducing build times can save many developers time on the daily. There are several different approaches I've seen studios take to optimize build time. Here are a few of them:
Instead of having everybody build everything, we separate assets into buckets that don't always need to be built - common assets in one bucket, specific level assets in another, character assets in a third, weapons in a fourth, and so on. This way, a character artist might need to build the character bucket but is fine with slightly out-of-date level or audio assets. A level designer might be building their assigned level all the time, but doesn't need to build all of the other stuff. And so on. If the build is modular, I only have to build the assets I need to rather than all of them.
Getting more computer hardware is relatively cheap, especially if there are a lot of people on the team. Build engineers set up and maintain workstations whose only job is to grab the latest data from the depot, build the assets, then serve the finished files to developers who ask for them over the network. Automated builds also run automated tests on those builds to ensure they work. If the build is broken for some reason, the build machines are set up to look at all of the checkins between the current (broken) build and the last good build so they can warn the team that the build is broken, and identify and notify the culprits responsible for checking in broken files. Automated builds typically run 24/7, meaning that freshly built files are often only minutes or hours old at most, and that broken builds get caught more quickly so fewer devs get stuck.
Our development workstations are not always running at 100% capacity at all times. While I'm thinking, reading code, writing documentation, or generally not putting my PC through the paces, its CPU is mostly just idling and maybe playing some music for me. There are software suites that will allow these idle workstations to help out during a teammate's build process by doing some of the processing concurrently. Available idle workstations can get pulled in to help speed up a teammate's build. This also often works with the automated build principle - hardware is fairly cheap relative to developer time, so it's worth spending a few thousand for build machines to save collective days, weeks, months of wasted developer time.
Full Time Build Engineers
In order to make these things work, we hire engineers who work full time on making builds go faster. It is their job to set up and automate the build process for the rest of us. They might not work on game code or assets directly, but their work is invaluable to keeping the entire team productive. They set up the automated build process, they maintain the build machines, they create the automated tests, they handle the integrations between depots, and they are the first line of defense against bad checkins.
These are the most common ways I've seen of speeding build times up and keeping us productive. Optimizing builds is a never-ending battle where the victories can save thousands of collective hours, and the defeats can cost thousands of collective hours. It's an enormous problem that grows multiplicatively with the size of the team, because every additional team member is more time lost to building.
[Join us on Discord] and/or [Support us on Patreon]
Got a burning question you want answered?
Short questions: Ask a Game Dev on Twitter
Long questions: Ask a Game Dev on Tumblr
Frequent Questions: The FAQ
