#build systems

Every major studio I've worked at generally has their own automated build system. These build systems are not shared between projects at the same publisher. This is because each game has its own file structure, asset list, and scripts that must be processed alongside building the code in order to result in a functioning game build. The larger the project/franchise, the more elaborate and powerful these build systems tend to be.

The Call of Duty team, for example, has a build system that allows developers to build the game with a shelved changelist against the current depot on multiple platforms (PS4, PS5, XB1, XSX, PC), and run a full suite of tests on the result in order to validate the changes. The devs are required to run this build and validation process before being allowed to submit their changes in order to ensure stability in the game. The build system is also constantly running in a cycle to compile all of the game's maps and packages for QA to test.

[Join us on Discord] and/or [Support us on Patreon]

Got a burning question you want answered?

Short questions: Ask a Game Dev on Twitter

Long questions: Ask a Game Dev on Tumblr

It's good to remember what a vertical slice is - it's a functioning version of all of the systems we plan on having in the game working together so that it is representative of the final game experience. MMORPGs tend to come with a lot of expected table stakes features, which means our vertical slice needs to have those features. For a MMORPG, that generally entails:

At least two distinct players on separate accounts able to log in and play in the same game world

A persistent game world - all players are able to log out and the same game state persists between logins

Functioning gameplay systems - skills, targeting, combat, leveling, crafting, etc.

MOBs that exist within the world independent of players

At least one quest that the players are able to accept, progress, complete, and obtain rewards

A functioning leveling system that allows players to level up and obtain rewards

Functional UI - combat, looting, character inventory, NPC interaction (e.g. quests, cinematic), chat, grouping with other players, etc.

A functioning chat system - player A can type a message and player B can read it in real time

A functioning item storage system (e.g. bank)

A functioning asynchronous mail system - player A can send a mail to player B who can receive it when they next log in.

A functioning player-to-player commerce system (auction house)

Functioning player-to-player direct trading

A functioning gear system

For a vertical slice to be a vertical slice, we don't need all of the different classes or races or gear or systems, but we need at least one of everything working together so that the play experience of the vertical slice is representative of the finished game play experience. Core functionality all has to be there. It can be a little unstable and it can be cobbled together, but everything should be there and it needs to feel like the intended final experience. The purpose of the vertical slice is to persuade the executives that the rest of the game is worth making.

[Join us on Discord] and/or [Support us on Patreon]

Got a burning question you want answered?

Short questions: Ask a Game Dev on Twitter

Long questions: Ask a Game Dev on Tumblr

The larger the team gets, the more effort and resources the studio spends on reducing build times. This is because it's a multiplicative effect - reducing build times can save many developers time on the daily. There are several different approaches I've seen studios take to optimize build time. Here are a few of them:

Modular Builds

Instead of having everybody build everything, we separate assets into buckets that don't always need to be built - common assets in one bucket, specific level assets in another, character assets in a third, weapons in a fourth, and so on. This way, a character artist might need to build the character bucket but is fine with slightly out-of-date level or audio assets. A level designer might be building their assigned level all the time, but doesn't need to build all of the other stuff. And so on. If the build is modular, I only have to build the assets I need to rather than all of them.

Automated Builds

Getting more computer hardware is relatively cheap, especially if there are a lot of people on the team. Build engineers set up and maintain workstations whose only job is to grab the latest data from the depot, build the assets, then serve the finished files to developers who ask for them over the network. Automated builds also run automated tests on those builds to ensure they work. If the build is broken for some reason, the build machines are set up to look at all of the checkins between the current (broken) build and the last good build so they can warn the team that the build is broken, and identify and notify the culprits responsible for checking in broken files. Automated builds typically run 24/7, meaning that freshly built files are often only minutes or hours old at most, and that broken builds get caught more quickly so fewer devs get stuck.

Distributed Builds

Our development workstations are not always running at 100% capacity at all times. While I'm thinking, reading code, writing documentation, or generally not putting my PC through the paces, its CPU is mostly just idling and maybe playing some music for me. There are software suites that will allow these idle workstations to help out during a teammate's build process by doing some of the processing concurrently. Available idle workstations can get pulled in to help speed up a teammate's build. This also often works with the automated build principle - hardware is fairly cheap relative to developer time, so it's worth spending a few thousand for build machines to save collective days, weeks, months of wasted developer time.

Full Time Build Engineers

In order to make these things work, we hire engineers who work full time on making builds go faster. It is their job to set up and automate the build process for the rest of us. They might not work on game code or assets directly, but their work is invaluable to keeping the entire team productive. They set up the automated build process, they maintain the build machines, they create the automated tests, they handle the integrations between depots, and they are the first line of defense against bad checkins.

These are the most common ways I've seen of speeding build times up and keeping us productive. Optimizing builds is a never-ending battle where the victories can save thousands of collective hours, and the defeats can cost thousands of collective hours. It's an enormous problem that grows multiplicatively with the size of the team, because every additional team member is more time lost to building.

[Join us on Discord] and/or [Support us on Patreon]

Got a burning question you want answered?

Short questions: Ask a Game Dev on Twitter

Long questions: Ask a Game Dev on Tumblr

Buck2 is a powerful, open-source build system that Facebook uses to rapidly compile and test our software. Originally developed internally, it’s now available for the world to use, offering significant advantages in speed and efficiency compared to traditional build systems. This guide will explore what Buck2 is, why it’s gaining traction, and how you can start using it. What is Buck2? Buck2 was…

While we're at it it would be nice to set the builder up in such a way that it can itself hot-reload, which feels like an unsolvable bootstrapping problem??

I guess I can move the build logic into a data file, which can reload without needing a code update, and that covers everything sane I could do with a hot-reloading code-rebuilder

I tried, and failed, to do this. I could compile a certain version of CPython, but some of the dependencies of the latest CPython gave me trouble. I could compile PyGame with a simple makefile, but it was more difficult with meson.

Instead of doing this by hand, I started to write a Dockerfile. It's just too easy to get this wrong otherwise, or at least not get it right in a reproducible fashion. Although everything I was doing was just statically compiled, and it should all have worked with a shell script, it didn't work with a shell script in practice, because cmake, meson, and autotools all leak bits and pieces of my host system into the final product. Some things, like libGL, should never be linked into or distributed with my executable.

I also thought that, if I was already working with static compilation, I could just link PyGame-CE against cosmopolitan libc, and have the SDL2 pieces replaced with a dynamically linked libSDL2 for the target platform.

I ran into some trouble. I asked for help online.

The first answer I got was "You should just use PyInstaller for deployment"

The second answer was "You should use Docker for application deployment. Just start with

FROM python:3.11

and go from there"

The others agreed. I couldn't get through to them.

It's the perfect example of Docker users seeing Docker as the solution for everything, even when I was already using Docker (actually Podman).

I think in the long run, Docker has already caused, and will continue to cause, these problems:

Over-reliance on containerisation is slowly making build processes, dependencies, and deployment more brittle than necessary, because it still works in Docker

Over-reliance on containerisation is making the actual build process outside of a container or even in a container based on a different image more painful, as well as multi-stage build processes when dependencies want to be built in their own containers