#game certification

I’m still crunching so it is still difficult for me to find the time for full length posts, but I’ve gotten a number of questions about Cyberpunk 2077 getting delayed again after going gold and it’s a pretty quick answer.

The cert process generally has two sets of criteria - deal breakers that fail outright (e.g. crashing, using incorrect platform trademarks like button names/icons, etc.), and things that don’t cause outright failure but are worth a point value. If a game accumulates too many points, it also results in cert failure. However, if the game has few enough points, it can pass cert with no required changes. 

There’s also window between a full pass and cert failure called “conditional pass” which is not quite bad enough to fail, but still bad enough to require fixes. Conditional passes come with a list of must-fix issues in a day 1 patch. No day 1 patch means no approval. What almost certainly happened was that Cyberpunk 2077 got a conditional cert pass due to enough issues that weren’t bad enough to fail cert outright but also weren’t benign enough to get a full pass. This was their “going gold” status - they got the conditional pass from cert.

Longtime readers should know that patches also require certification and can also fail... which is almost certainly what happened in this case. CDPR most likely failed cert for their day 1 patch and thus must continue to crunch to try to fix it. They probably did not expect to fail cert (again), but they also cannot release the game without passing cert, so they had to delay the game (again) despite “going gold”. My suspicions are that they devs are probably running on fumes thanks to having crunched for a year and a half already.

[Join us on Discord] and/or [Support us on Patreon]

The FANTa Project is being rebooted. [What is the FANTa project?]

Got a burning question you want answered?

Short questions: Ask a Game Dev on Twitter

Long questions: Ask a Game Dev on Tumblr

It’s the source of funding that typically decides whether a platform will be supported because they are the ones who have to decide whether to spend the money to support that platform. Most of the time, that’s the publisher. The decision to support last-gen hardware is typically a question of cost compared to estimated revenue. The publishers (with our team leadership’s input) weigh the development cost it will take to build the game with a reasonable play experience for any given platform and compare it to their revenue estimates for that platform. If they think there is sufficient money to be made from developing for, porting to, or building a smaller-scope separate game for that platform, they’ll do it.

Publishers usually continue to support last-gen consoles for about a year into the lifespan of the new console generation. This is typically because most of these games started development during the prior generation. Once the technical differences become untenable to maintain (e.g. GPU texture processing, CPU performance, amount of system RAM, etc.) within the same project, the call comes from above that the old platforms have been cut loose. This is primarily because the game’s development team would need to fork into two separate teams by this point - one to build a game for each platform. Most of the time, the last-gen version of the game won’t be expected to earn enough to justify that kind of resource expenditure, so the support is dropped. In some rare cases, the installed based for the platform is so huge that building a completely separate game for the platform is worth it - the 3DS Madden NFL game in 2011 was built completely separately from its PS3/X360 console counterparts by a completely separate team. In other rare cases when a new platform just doesn’t have an audience, the publishers will quickly drop the new platform as well (e.g. Wii U). 

Most console manufacturers are generally happy to certify games for old consoles for publishers that want to pay for the cert process. As some examples of this, Just Dance 2020 is certified for the Wii. Persona 4 came out for the Playstation 2 in 2008, two years after the PS3 launched. Some of them will eventually stop supporting their own platforms after sufficient time has passed, but that’s typically long after publishers stop making games for them. You can usually spot this when retailers stop stocking the old consoles.

[Join us on Discord] and/or [Support us on Patreon]

The FANTa Project is being rebooted. [What is the FANTa project?]

Got a burning question you want answered?

Short questions: Ask a Game Dev on Twitter

Long questions: Ask a Game Dev on Tumblr

Honestly, it’s usually because we’re busy trying to make the game work on all of the platforms we’re shipping on. The main thought is this - we have a limited amount of development time and resources (i.e. “zots”) we can spend to build the game. If we have 100 zots to spend, do we want to spend them developing a feature that is only supported by one platform or all four platforms? This restriction applies to anything that’s platform-specific, like a touchpad, motion or gyroscopic controls, those weird SPUs on the PS3, and so on. 

This is why platform exclusives tend to take more advantage of the unique elements of the platform. We know that the return on our zot investment will be good - everybody who plays the game will be on the same platform, so we can invest more zots into the stuff unique to the platform. We aren’t spending our 100 zots on a feature that will only be used by 25% of the audience, we’re spending them on a platform-exclusive game it will be used by 100% of the audience.

That said, some platforms will include “exclusive content” in their certification requirements, often to make it a selling point to customers. This means we on the dev side must include something that is unique to the platform if we want to release on that platform. Unfortunately, this generally means we spend as few zots on the exclusive content as possible for the mentioned efficiency reasons - we want to get the biggest bang out of our zots that we can. For a cross-platform game, that means to spend our zots on things usable across as many different platforms as we can so the features/content we create with those zots can reach the maximum numbers of players.

[Join us on Discord] and/or [Support us on Patreon]

The FANTa Project is being rebooted. [What is the FANTa project?]

Got a burning question you want answered?

Short questions: Ask a Game Dev on Twitter

Long questions: Ask a Game Dev on Tumblr

The situation you describe isn’t actually soft locking. What you’re describing is the player getting, to use the highly technical term, stuck. That said, there are a lot of possible situations where the player achieves a state where they cannot play the game. While the overall result might look similar, the causes are usually very different and thus require different sorts of solutions.

#1: Freeze or Hard Lock

This occurs when all visible activity for the game stops - no more screen updates, input doesn’t work, nothing at all. Hard locks usually occur when the game has crashed and can’t clean up the rest of the program. Instead, it just sits there in system memory, forever unchanged until you reset the device or externally terminate it. The solution to this is to find the code that was executing while it crashed and either fix the code itself so it doesn’t crash, or fix the circumstances that caused the code to crash (e.g. invalid asset data).

#2: Soft Lock

A soft lock occurs when the game is unresponsive and unplayable, but the screen continues to update. Animations still play, sound still plays, UI still updates, timers may still tick down, but the game will not respond to any player input. This is usually caused by individual systems in the game turning player control/input off in order to make sure that they don’t do weird stuff during transitions (e.g. after entering a new zone but before the zone is fully loaded and ready, or not letting players move their characters in the background while opening up a UI menu) and then missing a case to turn it back on again. The solution to soft locks like this is usually to figure out what was locking player input and finding the corresponding case where control should be returned to the player.

#3: Stuck

Sometimes, when navigating the game environments, players will get stuck on things. This means that they’ll be trapped in the level somewhere and be unable to move. Maybe that’s because the world geometry has put them into a perpetually-falling state so they don’t have any control, or because they’ve somehow clipped into a part of the level that should not be accessible. Large 3D environments are especially known for this problem occurring quite often. This also applies to players falling out of the world or getting stuck outside of the world (e.g. behind walls) where they might be able to move, but cannot play the game. We have a couple of ways to solve this problem. Obviously, the first fix is to have level design fix the offending level geometry so that the player can’t get stuck there anymore. However, this can become problematic in games with really large environments, because testing every square inch of every environment in an open-world game like Horizon Zero Dawn or Breath of the Wild is a monstrous task for QA. Often we have a secondary solution for getting stuck in games like this - we try to store the last known “good” position for the player and then teleport the player to that position if they get stuck. If that doesn’t work (e.g. the last known good position isn’t so good), we can teleport the player to their respawn location as if they had died.

#4: Broken

Often in games the player is required to accomplish certain tasks or acquire certain items to progress. If those tasks are not marked as complete or those items are not obtainable, the player is barred from progressing. In this case, we call it a broken state - the game can be played but cannot be completed. Something necessary for the player to continue in the game is somehow unobtainable. The solution to this is usually to figure out what the player is missing from the criteria and track down what was supposed to grant the missing element to the player. It’s usually something small, like a bad set of conditions, an improper quest trigger, or just a misspelling of a particular quest variable name.

Each of these kinds of problems will usually cause our game to fail certification (though some small amount of broken content is generally acceptable if they are small enough and aren’t on the game’s critical path) if discovered, so they are all pretty high priority fixes when we find them. Each type of fail state generally falls to different disciplines on the team - freezes and soft locks are usually handled by programmers while stuck or broken bugs are often handled by designers due to their common causes. 

[Join us on Discord]

The FANTa Project is currently on hiatus while I am crunching at work too busy.

[What is the FANTa project?] [Git the FANTa Project]

Got a burning question you want answered?

Short questions: Ask a Game Dev on Twitter

Long questions: Ask a Game Dev on Tumblr

Games were never released “just right”, even back then. Games are (and were) always deemed “good enough” to release, i.e. passing certification requirements. We had to meet certification requirements and a quality bar back then, just like we have now. The certification and quality requirements are not really any different then than they are now.  There were plenty of buggy mess games back then, just like we have now. Take a look at any speed-running game community and you’ll see how often exploitable bugs appeared in all sorts of pre-internet games.

The main difference between then and now is that we can patch and change the game after the submission process. Before, we either passed or failed certification. If we passed cert, that was it - pencils down, no more changes allowed at all. If we failed cert, we’d fix exactly what failed the cert process and no more. Repeat that process until we pass and then we’re done with this particular game forever. This meant we had to be a lot more cautious about what was included in the game. Content and features that were almost finished but would probably not pass cert had to be cut without a second thought. 

Thanks to the internet, the certification process now has three outcomes, not two. In addition to pass and fail, there is now a “contingent pass” status, where they give us a pass on the condition that we fix these particular (lesser) issues by launch in a day zero patch. This gives us some additional post-submission leeway time after the cert process to fix the necessary problems and complete the game. This means that we can rebudget some of the time (which used to be spent before cert removing the content that was almost good enough) to complete that almost-good-enough content before cert and get it over the finish line. 

Basically, the ability to patch allows us to finish more overall game content that would have been cut before. The perception of increased bugginess is mostly in your mind, not reality. Games were just as buggy before, if not buggier. Players just didn’t notice as much back then because there wasn’t an internet infrastructure for them to share that stuff so rapidly.

[Join us on Discord]

The FANTa Project is currently on hiatus while I am crunching at work too busy.

[What is the FANTa project?] [Git the FANTa Project]

Got a burning question you want answered?

Short questions: Ask a Game Dev on Twitter

Long questions: Ask a Game Dev on Tumblr

There are some pretty big benefits to having an established platform like that. Players have become accustomed to a certain set of features - friend lists, the ability to invite friends to play together, sending messages back and forth, automatic patch updates and DLC installation, achievements, etc. Somebody has to build and maintain all that, and that takes time, resources, and a commitment to maintenance.

The biggest benefit to third party publishers is that they don’t have to spend a whole bunch of resources building all that stuff for themselves. If we want to patch a game, we build a working patch and send it to Nintendo, who certifies it and then their system handles the distribution and installation to all of the players on the platform. It’s on Nintendo to make sure all of the players are running the most recent version of the game, that players can invite each other to play, etc. 

Building and maintaining an entire platform can be really expensive - we need to keep track of a lot of user data, financial data, and do a lot of work on the back end making sure that it’s secure, reliable, and running 24/7. To do all of that, you need UI/UX designers and artists, server programmers, database programmers, security specialists, database administrators, and a 24/7 maintenance crew to keep everything running.It makes a lot more sense for the platform guys to do it themselves, since they also get to keep everything on the platform consistent and all of the games running on that platform can reuse the same technology and features that are provided. It keeps the user base from getting fragmented (i.e. you don’t separate the players between an EA launcher, a UPlay launcher, a Steam launcher, etc.).

The FANTa Project is currently on hiatus while I am crunching at work too busy.

[What is the FANTa project?] [Git the FANTa Project]

Got a burning question you want answered?

Short questions: Ask a Game Dev on Twitter

You should check out some of the links in the last post, [this one in particular covers this exact question]. Here’s the quick and dirty on Day 1 DLC:

All DLC has its own separate budget, including day 1 DLC. Day 1 DLC is typically stuff we started developing during production and cut because it didn’t fit in the schedule and wasn’t deemed high enough priority for the developers involved to include with the base game. It is usually close enough to being finished that it only needed a relatively small amount of additional work to get it over the finish line, engaging enough to make people want it, and modular enough that its absence would not ruin the game. It just didn’t make it into the base game because it wasn’t high enough priority compared to the other stuff we needed to finish in order to ship the game. But it is still paid for out of a separate budget, like all DLC. 

Javik is basically a textbook example of DLC content. He was originally intended to be one of the characters in the game, but was cut because he was the least important to the story and there was too much other work to finish him by the time the game shipped. Some people complain that Javik was essential to the story because he is a prothean, but they are wrong. There are plenty of problems with Mass Effect 3 as it is. Making Javik part of the core cast instead of one of the others would have fixed precisely none of them. 

Some people also complain about how some of Javik’s files were included on the disc, and that this means they should be entitled to him. His model and base animations were on the disc, yes, but that doesn’t mean that the rest of his associated content was on the disc or that it was tuned and ready for players. Work-in-progress locked-off content ships on discs all the time. GTA3 had Hot coffee, KOTOR 2 had a bunch of extended ending cinematics, Persona 5 had the Persona 3 lounge modeled but left untextured, Final Fantasy 15 had swimming animations for Noctis left in even though there are no places players are allowed to swim in the game, and even World of Warcraft shipped with entire unfinished zones locked off (Silithus, Hyjal, Azshara). We do it because removing these assets can break game stability and cause unforeseen structural problems. So we leave this stuff in the code and assets with no way to activate them, like a fossil record of what might have been. Sometimes we get the resources to resurrect the fossils with DLC or expansion packs. That doesn’t mean players are somehow entitled to the finished versions of this content just because they have some of the unfinished assets. 

Usually Day 1 DLC is content that designers and artists work on during the dead period while we wait for [certification]. Designers and artists not allowed to touch the base game during the final weeks to keep from introducing new bugs that might fail the certification process. Only the engineers are, and even they are only allowed to fix cert-blocking bugs. So, rather than have these designers and artists idle during the weeks for games to pass, we either move them to new teams, or have them work on DLC in order to justify their paychecks. The DLC cert process is waaaay faster than game certification, which is why we can often get the DLC ready on day 1.

Got a burning question you want answered?

Short questions: Ask a Game Dev on Twitter

Long questions: Ask a Game Dev on Tumblr

I apologize, I skipped a step and it wasn’t clear to you. Here’s how the procedure goes: 

When we submit our game for certification (let’s call it AAGD’s Game version A) with Sony, the Sony guys take it in its entirety. If version A passes cert, it retains that status for as long as nothing in version A changes. If version A fails, we fix what’s wrong and proceed to submit version B, then C, then D, etc. until we pass.

Patches also get their submissions. Patch Alpha gets a version A, then a version B, etc. until it passes cert. Patch Alpha must be certified to work with our game’s certified version A, because AAGD’s Game version A is what’s certified. Sony tests and Patch Alpha version B passes. At this point, we have two certified products for the platform: AAGD’s Game version A, and patch Alpha version B. As we continue to support the game, these certified patches can accumulate. We could end up with AAGD’s Game version A, patch Alpha version B, patch Beta version G, patch Gamma version L, and so on and so forth. All certified.

If we want to release a GOTY bundle, however, we cannot just patch the game up to the latest patch without having to run the fully patched version through certification again as AAGD’s Slightly Altered Game version A. Why? Because Sony has no guarantee that we’re only applying previously certified patches aside from our word. We might have slipped in additional stuff that breaks certification requirements. Sony must verify all of the contents of the new version of the game. If we just pack in the original AAGD’s Game version A, Patch Alpha version B, patch Beta version G, patch Gamma version L, etc. (all certified) and require the player to assemble them herself, then all we are doing is including a collection of separate items that have all passed certification already.

If we are porting AAGD’s Game to a new platform (e.g. the Switch) and include all of the postlaunch content and patches, we can submit AAGD’s GOTY Game to Nintendo as a package and go through cert. Switch players of AAGD’s GOTY Game will get the full bundle without needing to patch. But just because AAGD’s GOTY Game went through Switch cert and passed doesn’t mean Sony is cool with it. We can’t release AAGD’s GOTY Game on PS4 without submitting it for certification with Sony first. Microsoft is the same. 

This is what I meant by “one certification is cheaper than three”. If we are porting a title to a new platform, we must go through (and pay for) certification on that platform. That’s the “one certification”. But if we’ve already passed cert on the other platforms, we probably don’t pay thousands of dollars to submit the game again for package certification when all of the individual parts have already passed. We can just give you the already officially approved parts, and leave it to you to assemble them. It’s kind of a pain for new players, but the alternative is really expensive. Nobody likes going through cert (especially for an entire game), it’s an expensive, time-consuming process.

The FANTa Project is currently on hiatus while I am crunching at work too busy.

[What is the FANTa project?] [Git the FANTa Project]

Got a burning question you want answered?

Short questions: Ask a Game Dev on Twitter

Long questions: Ask a Game Dev on Tumblr