#fixing bugs

Recently i got into rust again. Partially for fun, partially to learn, partially because i always liked rust and ideas behind it, partially because it will soon be of use to me. I've been writing some code in it, learning to fight the borrow checker and the whole data-oriented mindset.

Today i was bored, so i decided to write some simple unit tests for my project, using the built-in features of cargo test and #[cfg(test)] etc.

Turns out i had a MAJOR bug in my program logic which caused my data structures to lose anywhere between 1 entry to 99% of stored data after an insertion operation. It somehow went unnoticed in all prior development, despite the fact that i used the library in some example executables.

The bug couldn't have been found by the compiler, since it was a program logic bug - no memory safety was harmed, all memory operations were perfectly safe. They just happened to drop references to a bunch of data nodes LMAO.

I was very unlikely to find the bug on my own, since - as i've said - i was sure the logic was working. All the programs i wrote seemed to return correct results.

What really struck me is that everyone talks about testing, and i have tried unit testing my projects in the past, be they in C, C++ or whatever, but i never got into the habit of doing it since it usually felt boring and never really provided me with much benefit - a minor bug there, an off by one here, a missing if statement somewhere else.

This, however, was the first time where a bug happened that i was very much unlikely to catch without unit tests, and said bug was actually a major one that would completely nullify usefulness of my project! I have never in the past experienced such direct and significant benefit from writing a few test cases.

It also felt funny in a way, since that's specifically what testing is for. Hell, i work in software testing (sort-of). Despite that, i usually let the message of TDD preachers fall on deaf ears. I guess i will not be doing that anymore lmao.

I felt very glad that i wrote the tests. In fact, fixing that bug also made me rethink the code i was using in that particular place, and realised it was VERY inefficient (i wrote it this way since borrow checker was being mean to me and i was just glad once the code seemed to work lmao). I learned a new standard library function that allowed me to write not only correct, but also more efficient code!

Afterwards i installed llvm-cov cargo addon to check my coverage, and noticed that some of the more complex branches of code were missing from it. This in turn made me write more test cases.

In the end i was so hyped up (by testing, of all things, lmao) that i added CI to my repo and can now enjoy a green checkmark next to my commits and log that says "all tests passing" :)

Yeah, it's pretty normal. These kind of bugs commonly come in via playthroughs during a feature review (usually producers, sometimes designers/engineers/art/QA, sometimes bigwigs), and often have only the comments made during the review for context. It usually requires some analysis after the fact, optimally with video footage of the playthrough to help provide some context to the issue. These kind of issues often have to go through production and get reviewed by the subject matter experts to interpret what is actually being asked. Once that's figured out, they get assigned. The bad thing is that these bugs are often difficult to reproduce consistently by QA (or they'd have been written up by QA already), but the behavior happens often enough that the bug is still a must-fix.

In that kind of situation, a hard-to-reproduce bug is often tasked to an engineer with a solid understanding of the systems who can track the bad behavior through the code, script, and assets to figure out what's causing it. These engineers aren't necessarily responsible for fixing the bug, but they are the specialists who figure out the cause of the bad behavior (e.g. animation X got checked in with a bad export) so that someone on the team can fix it (e.g. animator re-exports animation X).

[Join us on Discord] and/or [Support us on Patreon]

Got a burning question you want answered?

Short questions: Ask a Game Dev on Twitter

Short questions: Ask a Game Dev on BlueSky

Long questions: Ask a Game Dev on Tumblr

Entering 2021 strong with more knowledge and more experience.

2020 was a year of learning. It taught us the value of time, to believe in ourselves, to make the best use of what we have, not take anything for granted and always keep learning. 

Only then we can overcome our biggest fears.

The situation you describe isn’t actually soft locking. What you’re describing is the player getting, to use the highly technical term, stuck. That said, there are a lot of possible situations where the player achieves a state where they cannot play the game. While the overall result might look similar, the causes are usually very different and thus require different sorts of solutions.

#1: Freeze or Hard Lock

This occurs when all visible activity for the game stops - no more screen updates, input doesn’t work, nothing at all. Hard locks usually occur when the game has crashed and can’t clean up the rest of the program. Instead, it just sits there in system memory, forever unchanged until you reset the device or externally terminate it. The solution to this is to find the code that was executing while it crashed and either fix the code itself so it doesn’t crash, or fix the circumstances that caused the code to crash (e.g. invalid asset data).

#2: Soft Lock

A soft lock occurs when the game is unresponsive and unplayable, but the screen continues to update. Animations still play, sound still plays, UI still updates, timers may still tick down, but the game will not respond to any player input. This is usually caused by individual systems in the game turning player control/input off in order to make sure that they don’t do weird stuff during transitions (e.g. after entering a new zone but before the zone is fully loaded and ready, or not letting players move their characters in the background while opening up a UI menu) and then missing a case to turn it back on again. The solution to soft locks like this is usually to figure out what was locking player input and finding the corresponding case where control should be returned to the player.

#3: Stuck

Sometimes, when navigating the game environments, players will get stuck on things. This means that they’ll be trapped in the level somewhere and be unable to move. Maybe that’s because the world geometry has put them into a perpetually-falling state so they don’t have any control, or because they’ve somehow clipped into a part of the level that should not be accessible. Large 3D environments are especially known for this problem occurring quite often. This also applies to players falling out of the world or getting stuck outside of the world (e.g. behind walls) where they might be able to move, but cannot play the game. We have a couple of ways to solve this problem. Obviously, the first fix is to have level design fix the offending level geometry so that the player can’t get stuck there anymore. However, this can become problematic in games with really large environments, because testing every square inch of every environment in an open-world game like Horizon Zero Dawn or Breath of the Wild is a monstrous task for QA. Often we have a secondary solution for getting stuck in games like this - we try to store the last known “good” position for the player and then teleport the player to that position if they get stuck. If that doesn’t work (e.g. the last known good position isn’t so good), we can teleport the player to their respawn location as if they had died.

#4: Broken

Often in games the player is required to accomplish certain tasks or acquire certain items to progress. If those tasks are not marked as complete or those items are not obtainable, the player is barred from progressing. In this case, we call it a broken state - the game can be played but cannot be completed. Something necessary for the player to continue in the game is somehow unobtainable. The solution to this is usually to figure out what the player is missing from the criteria and track down what was supposed to grant the missing element to the player. It’s usually something small, like a bad set of conditions, an improper quest trigger, or just a misspelling of a particular quest variable name.

Each of these kinds of problems will usually cause our game to fail certification (though some small amount of broken content is generally acceptable if they are small enough and aren’t on the game’s critical path) if discovered, so they are all pretty high priority fixes when we find them. Each type of fail state generally falls to different disciplines on the team - freezes and soft locks are usually handled by programmers while stuck or broken bugs are often handled by designers due to their common causes. 

[Join us on Discord]

The FANTa Project is currently on hiatus while I am crunching at work too busy.

[What is the FANTa project?] [Git the FANTa Project]

Got a burning question you want answered?

Short questions: Ask a Game Dev on Twitter

Long questions: Ask a Game Dev on Tumblr

There’s an industry term for what you’re asking about here - it’s called Technical Debt. Whenever you make a decision on a system that will require fixing or maintenance at some point in the future, it is incurring tech debt that must be paid later in the form of developer time, resources, etc. Putting in a temporary X just to make something work (e.g. make something work for E3) means that, at some point in the future, you need to replace the temporary X we put in with a permanent X, or suffer the drawbacks of the temporary X forever. That total amount of effort of replacing the temporary X with the permanent X is tech debt.

Like most other forms of debt, there’s a minimum payment for tech debt. This minimum payment is some amount of developmental resources that must be paid in order to keep the project moving forward. It could be something small like “don’t ever using this specific combination of options or the game will crash”, or it could be something large like “the process works fine for incremental builds but a full build takes sixteen hours to complete”, but it’s something the dev team must put up with in order to keep things going. If somebody forgets one day and doesn’t follow the weird requirements, it can end up breaking the level until it’s fixed (or requiring a sixteen hour full rebuild) for all of the devs working on that level, causing a lot of lost productivity. This additional constraint/requirement and potential loss of productivity is the minimum payment the team pays for incurring the tech debt.

Sometimes dealing with the tech debt is just too severe to deal with long term (e.g. a sixteen hour build time), so we seek to pay that tech debt off. This requires more development resources to fix the problem than the minimum, and the fix may incur additional tech debt of its own (though probably of a different nature than before). This isn’t necessarily proportionate to the minimum payment either; sometimes a full tech debt payment is exorbitant compared to the minimum payment. For example, I once discovered a bug in the level editor where the locations of trigger volumes that were rotated and scaled in different directions would be inaccurate when running the actual game. I’d place the volume in this place in the editor, but it would not actually be at that location in the game. I asked about this and one of the engineers looked into it, but declared that the problem was pretty deep in the code to handle the export process. Fixing this would require a lot of work and could have a lot of potential negative side effects because it affected the way we handled all 3D object placement and orientation in the game world. Because of the cost involved with paying off the tech debt, it’s a value judgement - is the benefit of clearing up the tech debt worth more to us long-term than the effort we’d need to spend for the rest of the project just handling the minimum payment? In this particular case, I promised that the designers wouldn’t ever use rotated trigger volumes because the minimum payment (not using rotated volumes) was cheaper than paying off the debt entirely (fixing the underlying problem).

You also have to keep in mind that paying off tech debt does not come from its own budget. New feature and content development also pulls from the same resource pool because it’s the same developers who are building new features and content. If I’m fixing old problems, I’m not building a new dungeon or creating new items or tuning a new monster. Also, you need to keep in mind that fixing tech debt often doesn’t garner the same reaction from the player base - new features get people to keep playing, while many of the tech debt issues (like reducing a sixteen hour build time) is not immediately felt by the players. This means that decision-makers often need to triage the tasks - is paying off the tech debt worth spending our resources on or should we build new content with those resources instead? Sometimes the answer is yes and sometimes the answer is no, depending on the costs and circumstances involved.

[Join us on Discord]

The FANTa Project is currently on hiatus while I am crunching at work too busy.

[What is the FANTa project?] [Git the FANTa Project]

Got a burning question you want answered?

Short questions: Ask a Game Dev on Twitter

Long questions: Ask a Game Dev on Tumblr

There are lots of instances of stuff like this going on during game development. This is often because we can’t build things sequentially due to the length of time it takes to build stuff. For example, the game’s writers cannot decide to drop an ice dungeon storyline and take the story to a swamp instead even if it would make a better story, because we’ve already been building the ice dungeon assets and geometry for three months and we don’t want to throw that away in order to build a swamp. The writers have to come up with a way to get the players to the ice dungeon within their budgets and such.

Sometimes we have these incongruencies because of reasons beyond our control. Maybe we can’t bring the original actor back into the studio to record new lines, so we have to rewrite the dialogue around it in order to make the already-recorded lines work. Maybe the engine can only display 15 animated characters at a time without running into frame rate issues, so we do our best with the 15 characters we have to work with by using clever camera angles and cuts and such to make them seem like an army. Maybe we can’t fix the error in technology for this one scene, so we cover up the jitter with a screen shake.

Some of my more amusing bugs came from these incongruencies, like [changing the PC settings at the right time causing all in-game characters to be flat]. In one game I worked on, our face scan of one of the major movie characters came out really really badly and the actress didn’t have the room in her schedule for us to re-scan her face, so we could only show that character to the player from behind or the side. A lot of the experiences in games are often smoke and mirrors, cobbled together from barely-working stuff that would break if players could look at it in any other circumstance except the exact one we present. But that’s acceptable, because we can (try to) control those circumstances and present a fun experience for the player.

[Join us on Discord]

The FANTa Project is currently on hiatus while I am crunching at work too busy.

[What is the FANTa project?] [Git the FANTa Project]

Got a burning question you want answered?

Short questions: Ask a Game Dev on Twitter

Long questions: Ask a Game Dev on Tumblr