#technical debt

It has to do with how software engineering works. Whenever we build a system, that system doesn't typically exist in a vacuum - temporary or not, it exists to handle some particular task or solve some problem. A temporary solution might not handle that task or problem optimally, but it works well enough that the system can function within the greater context of all of the other systems that use it to handle their own tasks/problems.

If enough other systems are built on top of the temporary solution, that temporary solution becomes load bearing - removing that temporary solution will cause everything built upon it to break. Any attempt at replacing the temporary solution needs not only to replace the solution itself, but also needs to ensure that every system built on top of it still functions properly. Software development tends to be this constant layering of system upon system upon system as the game comes together. This generally means that the longer it takes to replace a temporary solution, the more difficult and expensive it becomes to do so. We call this cost [technical debt].

Since games must eventually ship, team leadership must make judgement calls on what debt we fully pay off and what debt we just pay the maintenance cost on. That determination is often determined by how much effort it takes to fix the problem properly and how much dev time we have left. Fixing tech debt is often mostly invisible to the players and we need to weigh those costs against having the developers work on new features or content instead. The temporary solutions work, they just don't work very well. Improvements in performance or workflow may not translate directly to value-add for the player.

[Join us on Discord] and/or [Support us on Patreon]

Got a burning question you want answered?

Short questions: Ask a Game Dev on Twitter

Short questions: Ask a Game Dev on BlueSky

Long questions: Ask a Game Dev on Tumblr

I lied when I said I was going to work next on loading a 3-D model. Sorry, old habit! Actually, I went straight into refactoring. Let me explain...

The English Wikipedia defines refactoring as "the process of restructuring existing computer code . . . without changing its external behavior", which is fairly accurate, though lacking in motivation.

My back-of-mind definition would be "changes to code whose primary purpose is not to add features or solve issues, but to make the codebase easier to maintain".

Back when I worked for corporations, I got in the habit of never mentioning refactoring around anyone who wasn't a software developer. If my boss (or my boss's boss) knew I was making changes (and spending work hours) on something other than approved features or known issues, awkward questions would arise. (Like, do we have a billing code for that?)

Anyone who's worked intimately with a large software project knows that if changes are made only for features and issues, the project will accumulate "technical debt" that makes it difficult to maintain: hard to explain/learn/understand/remember how it works and hard to make changes without introducing bugs.

Both of today's refactorings focussed on the BaseApplication class, which became unwieldy weeks ago. Last night the source file for the class reached 1901 lines of Java code (not counting blanks, comments, and javadoc). I don't place a hard limit on lines of code in a class, but a file containing 1901 LoCs positively screams technical debt. It's especially painful these days, since I'm working on a laptop with a tiny screen and using a track pad instead of a mouse. (I spend lots of time scrolling back and forth, hunting for the lines I need to change.) Cramming as much as possible into a single file makes some sense for a tutorial, but I see the V-Sport project as something I'll be maintaining for many years.

First I split off all the code that deals with physical devices and put that in a new PhysicalDevice class. The change greatly clarified which properties of the physical device matter and how that information is accessed.

Then I split off all the code that deals with texture data into a new Texture class. The new class bundles up 3 related Vulkan resources and provides a clear lifecycle of create/use/destroy. I expect it to minimize duplication of code when the project transitions (sometime in the near future) from a single texture to multiple textures.

I don’t have any good sources of job description as of yet, but simply counting the requirements and estimating the amount of years to complete each (and averaging to a standard for the decade) should be a good methodology depending on how complete the job description is assumed to be. 

If we do a project in a rough and ready way, which is often what we can manage under the time and budget constraints we face, we will build up a "debt" we'll eventually have to pay back. Basically, if we do it fast, we'll later have to go back and rework or even replace the code to make it robust enough for the long haul, interoperate with other code that runs our site or simply be truly functional as opposed just barely doing what we need it to. There's no right or wrong answer; it's simply a management challenge to know when to lean one way or the other. But if you build up too much of this debt the problem can start to grow not in a linear but an exponential fashion, until the system begins to cave in on itself with internal decay, breakdowns of interoperability and emergent failures which grow from both.

This is a fairly good description of what the media is now wrongly defining as the GOP's 'Trump problem', only in this case the problem isn't programming debt. It's a build up of what we might call 'hate debt' and 'nonsense debt' that has been growing up for years.

This crystallized for me after the last GOP debate when Trump told Chris Cuomo in a post-debate interview that the IRS might be coming after him because he's a "strong Christian." Set aside for the moment how this unchurched libertine was able to rebrand himself as a "strong Christian." What about the preposterous claim that he is being persecuted by the IRS because he is a devout member of the country's dominant religion? Republicans simply aren't in any position to criticize this ludicrous claim because they have spent years telling their voters that this sort of thing happens all the time - to Christians, conservatives, everyone the liberals at the IRS hate.

...I always try to remind people that as observers of contemporary politics we ascribe far more power to political leaders than they actually exercise. They are more like riverine engineers. They can reinforce the banks, shift a river's direction a bit. In extraordinary cases they can even dam a river. But even that only regulates the flow of the water. It seldom stops it entirely. The deeper causes of the recent trends in the GOP go deep into the society and culture of the American right and American society generally. But Republican elected officials have increasingly coddled, exploited and in some cases - yes - spurred their voters' penchant for resentment, perceived persecution, apocalyptic thinking and generic nonsense.

As you said, taking things away from players generates really bad feelings from the most engaged players. Everything that has been released and obtained by players must be supported forever for those players for the remaining lifetime of the game. However, the passage of time is inevitable and the game eventually loses its sustainability over time as player attrition causes engagement to inevitably go down. Even the most entrenched players will continue to age and external life events will occur that pull them from the game - they’ll play other games, they’ll graduate from school, they’ll start careers, they’ll get married, they’ll have kids, and so on and so forth. For many of these players, the amount of money and time spent on a game eventually becomes much less important. When you’ve suddenly become a father and have to take care of the baby, the shiny new raid or game mode often becomes a much lower priority in the grand scheme of things. Even the longest running live service titles like World of Warcraft see significant player attrition over time. 

In addition to this, the passage of time causes stress on the technology. Certain assumptions made early in the project (often technical) eventually become untenable for a live service to change or manage, primarily because of how much stuff is built on top of those assumptions. This can be things like how the user database is set up, how the hardware is assumed to work, or how the development tools to create the content were built. Things that worked just fine when there were only a thousand different item IDs in the game to load may no longer work as well when there are a hundred thousand different item IDs in the game. The team working on new content doesn’t have room in the budget or schedule go back to upgrade all of the prior content or technology. 

Eventually, the combination of these two major factors - player attrition and outdated technology/design - and their associated ongoing costs dictate that no further development should occur and the game should be quietly allowed to die. However, there are still a lot of fond memories among players who played the game and still a lot of general interest in the IP. This is when we decide to build the sequel for it. We can throw out the old crufty bits and start fresh with modern tech and design paradigms while (hopefully) retaining the concepts and ideas that made the previous game good. A new game will also be able to pull in new players who were interested in the old game but either bounced off due to things like complexity built up over its years as well as old players who lapsed due to some of those aforementioned life changes. 

We don’t always get it right for a variety of reasons, but many publishers have been able to establish community migration even with entrenched communities. The annual new sports title rollover, for example, is commonplace. Fighting game franchises like Street Fighter, Mortal Kombat, Dead or Alive, and Tekken have generally been able to transition their communities to new game iterations. Even some FPS franchises like Call of Duty and Battlefield are (generally) able to make those transitions. Overcoming the bad feelings of abandoning sunk cost is definitely a thing, but there are definitely ways do it properly.

[Join us on Discord] and/or [Support us on Patreon]

The FANTa Project is being rebooted. [What is the FANTa project?]

Got a burning question you want answered?

Short questions: Ask a Game Dev on Twitter

Long questions: Ask a Game Dev on Tumblr

There’s an industry term for what you’re asking about here - it’s called Technical Debt. Whenever you make a decision on a system that will require fixing or maintenance at some point in the future, it is incurring tech debt that must be paid later in the form of developer time, resources, etc. Putting in a temporary X just to make something work (e.g. make something work for E3) means that, at some point in the future, you need to replace the temporary X we put in with a permanent X, or suffer the drawbacks of the temporary X forever. That total amount of effort of replacing the temporary X with the permanent X is tech debt.

Like most other forms of debt, there’s a minimum payment for tech debt. This minimum payment is some amount of developmental resources that must be paid in order to keep the project moving forward. It could be something small like “don’t ever using this specific combination of options or the game will crash”, or it could be something large like “the process works fine for incremental builds but a full build takes sixteen hours to complete”, but it’s something the dev team must put up with in order to keep things going. If somebody forgets one day and doesn’t follow the weird requirements, it can end up breaking the level until it’s fixed (or requiring a sixteen hour full rebuild) for all of the devs working on that level, causing a lot of lost productivity. This additional constraint/requirement and potential loss of productivity is the minimum payment the team pays for incurring the tech debt.

Sometimes dealing with the tech debt is just too severe to deal with long term (e.g. a sixteen hour build time), so we seek to pay that tech debt off. This requires more development resources to fix the problem than the minimum, and the fix may incur additional tech debt of its own (though probably of a different nature than before). This isn’t necessarily proportionate to the minimum payment either; sometimes a full tech debt payment is exorbitant compared to the minimum payment. For example, I once discovered a bug in the level editor where the locations of trigger volumes that were rotated and scaled in different directions would be inaccurate when running the actual game. I’d place the volume in this place in the editor, but it would not actually be at that location in the game. I asked about this and one of the engineers looked into it, but declared that the problem was pretty deep in the code to handle the export process. Fixing this would require a lot of work and could have a lot of potential negative side effects because it affected the way we handled all 3D object placement and orientation in the game world. Because of the cost involved with paying off the tech debt, it’s a value judgement - is the benefit of clearing up the tech debt worth more to us long-term than the effort we’d need to spend for the rest of the project just handling the minimum payment? In this particular case, I promised that the designers wouldn’t ever use rotated trigger volumes because the minimum payment (not using rotated volumes) was cheaper than paying off the debt entirely (fixing the underlying problem).

You also have to keep in mind that paying off tech debt does not come from its own budget. New feature and content development also pulls from the same resource pool because it’s the same developers who are building new features and content. If I’m fixing old problems, I’m not building a new dungeon or creating new items or tuning a new monster. Also, you need to keep in mind that fixing tech debt often doesn’t garner the same reaction from the player base - new features get people to keep playing, while many of the tech debt issues (like reducing a sixteen hour build time) is not immediately felt by the players. This means that decision-makers often need to triage the tasks - is paying off the tech debt worth spending our resources on or should we build new content with those resources instead? Sometimes the answer is yes and sometimes the answer is no, depending on the costs and circumstances involved.

[Join us on Discord]

The FANTa Project is currently on hiatus while I am crunching at work too busy.

[What is the FANTa project?] [Git the FANTa Project]

Got a burning question you want answered?

Short questions: Ask a Game Dev on Twitter

Long questions: Ask a Game Dev on Tumblr