Hiya! I'm curious about your experience with QA. In my current company, the QAs are pretty deeply embedded, know all there is to know about the engine, and majority has enough tech know-how to debug stuff at source code level. On the other hand, I've heard horror stories about QAs treated as basically outsource contractors and limited to blackboxing only. In your opinion, which approach is more common in the industry, or maybe there are differences between AAA and smaller companies? Which ones do you prefer, personally? Thank you!
This is a real potential rabbit hole of a topic, so I have enlisted longtime QA expert and manager [KynetykKnows] for his expertise on the topic. I have my own thoughts on the matter, but I think that firsthand experience is better than second. His answer is as follows:
The first thing to realize is that no two QA departments (particularly in gaming) are going to be exactly the same. The bulk of testing is, by its nature, a reactive discipline that comes in at the end of the development pipeline - you can't test something that isn't built yet. That means the testing has to accommodate whatever the needs of that project/pipeline/team/company happen to be. As a result, there are a range of different approaches from deeply integrated white box test engineers to no internal QA at all relying on third parties to black box test with limited direction.
Most places aren't going to have all white box testing. This would incur more cost than would make sense for a lot of teams. It's much more difficult to find candidates that fit the requirements, the candidates would cost more to hire, and so on. Much more often you'll have teams built of black box testers. This alone doesn't necessarily result in a horror show. In fact, if a tester has technical savvy and initiative, they can often get into grey box testing and possibly full white box testing on their own. The team wouldn't be built around this, but it can still be a value add for the team.
The horror show element of this has more to do with the respect that QA gets at the studio. If the studio holds respect for QA, then everything is fine. If QA isn't respected (i.e. is treated as an unskilled position that you just need a warm body for), you'll wind up with toxic, exploitative environments. QA can be an entry level position, but it isn't an unskilled position. QA skills aren't something you can easily demonstrate with education or experience alone. If team leaders don't understand that distinction, it is an avenue for toxicity to creep in.
In my experience the smaller the team the more thinly everyone is spread and the more valuable additional skills on each dev become. The team hiring also has to take the team structure into account and scale can impact that. For example, imagine QA finds a serious bug but can't find solid repro steps. It's happened more than once on different hardware on a couple of different builds so it's clearly not some fluke. If the team has 100 engineers, QA can present all the info available and say "Here, you figure it out, we're here if you need more" and move on. Engineering probably has enough resources to continue the investigation without needing or providing further context. On the other hand if a team has 3 engineers and 1 QA, then in that context it can make more sense for the engineer to show the tester the code and maybe the tester can figure it out. With this in mind, a small team may put a premium on finding the most skilled candidate available to do whatever might be needed, whereas a large team may not have the luxury to be so demanding and needs to maintain a higher head count. Notice how this comes back to unskilled vs entry level.
As for personal preference, the first thing I should point out is that I've been around long enough that a dismissive attitude to QA doesn't survive me. I'm either not going to work at a place that does that, prove that it isn't true, or (if coming in on the ground floor) establish that this isn't how QA is to be treated. In general, I think it makes sense for not to require engineering skills for QA. There's a lot of good testing skill talent that you will cut off if you require enough technical skill for full white box testing. Furthermore, there may be a conflicting mentality. QA is out to test, break, and point out problems, not to fix them. I don't trust when an engineer says "this should be fine". If a tester has put in a code change, then I'd want a DIFFERENT tester to review the change. There's a reason authors need editors. But that's just my perspective, and to go back to the top, this all has to run in the service of needs of the specific needs of the engineering team, the project, and the company as a whole.
Many thanks to KynetykKnows for his thorough and in-depth answer.Â
[Join us on Discord] and/or [Support us on Patreon]
The FANTa Project is being rebooted. [What is the FANTa project?]
Got a burning question you want answered?
Short questions: Ask a Game Dev on Twitter
Long questions: Ask a Game Dev on Tumblr
Frequent Questions: The FAQ
