Over 2 years ago we released a prototype of the game along with this trailer. After that we began full development of the game and have been travelling from convention to convention showing it off and getting feedback. We’ve been keeping those interested up to date as best we can but with many developers focused on the game. In the spirit of maintaining transparency and honesty, we’ll explain what has been going on behind the scenes. Alicorn Princess Blast is a volunteer project formed by a group of friends. It hadn't gained any serious traction until after the first few conventions prompting us to form our company. People have been increasingly interested over the years which prompted us to begin more serious development on the game. There have been frequent setbacks due to people leaving and joining, meaning that a lot of the elements of the game have shifted and changed over time. Something else that has happened over the last two years is that those on the team have both gained and lost jobs which they rely on as a means of maintaining their own lives. As a result of this the project pace has slowed to more of a light jog down from rigorous sprinting. We have found that it can sometimes be difficult to focus all of our creative efforts on fine tuning and finishing the game when the urgent matters of a teammate require their immediate attention.

The lack of consistent updates on all fronts is one of the areas affected by this constant shifting. The majority of the developers have their efforts and spare time focused on finishing all of the features of the game. There have been very few moments where we have the time to update our audience on the progress we've made much to our own disappointment. We apologize for this and are making an effort to create a way to keep you all up to date on what's happening with our project. The plan will be simple. We're going to commit to giving you all monthly updates through our current channels so that you all remain informed on the development of the game. We’re going to kick this off with a big update on what’s happened ever since that trailer got released two years ago.

We’re In Alpha!

Firstly, the game has reached alpha and has been in that state for a little bit. We’ve been going from convention to convention- our most recent outing being at BronyCan, over the last year or so. We’d like to thank everyone who gave us feedback. While not complete quite yet, we’ve begun crucial testing to refine the experience to make sure that it fits comfortably for everyone. We believe that if we get the experience just right in the first level then it will be much easier for us to develop the game. We’ve also been discussing setting up a system that would include pushing out weekly or nightly builds to the public. It’s at this time where we’re really going to need help from the general public to help make this game amazing! If you’d like to help us out with the play test, sign up for our mailing list by going to www.alicornprincessblast.com.

Adding To the Fun

First, what is the purpose of testing a game? In short, testing a game is an act of information gathering that allows you to better address the needs of the game as a software application. It gives you insight into what is working, what is failing, and what will prevent your players from even playing. When bringing testers into your team it is always important to realize that their job is strictly data gathering and you should keep the proper amount that lets them do that job.

The goal in all of this is attempting to release the game within a certain "Acceptance Criteria" that essential says: Of the issues we know about, only minor ones are left that barely affect gameplay. There is know such thing as a perfect game, there is only learning to make better games. So in the interest of that we make games that play well and only have minor nitpick issues so that the player can enjoy the entire experience.

Thus my general plan for testing is as follows:

Develop bug criteria and guidelines

Perform active and regular testing

Perform playtests with interested parties

Reach the Bug Acceptance ratio before launch

Each of these tasks deserves their own section, therefore:

Develop bug criteria and guidelines

Bug criteria is essentially meta information about issues discovered, and guidelines is what our plan is for assigning said meta information. In coming up with our meta information, we need data about the bug that is useful to all the parties viewing it. Description is obvious, but less obvious are things such as severity of bug, priority, system information, location and build. How about a short glossary?

Severity: Depending on your system, you may label bugs as A-D for severity or some other collection of terms (Blocker, Critical, Major, Minor). The most severe bugs will be the ones that are cause the application to crash or worse, the least severe will be almost always be cosmetic or event feedback for changes. 

Priority: This is where the producer or leads take a look at the bug and decide just how important it is that the bug get's addressed. Usually the priority closely matches the Severity with a little room for wiggling depending on what is happening at the time. If you have to get a functional build out for playtesting, you just might lower the severity of bugs on maps that won't be played in the interest of having the map that will be played function.

System Information: This is part of some general housekeeping for every bug where in the tester provides as much information as possible about the environment that this occurred in. Smart testers will be able to recognize when certain information is important, but it's become expected that system information is always important.

Location: Otherwise known as: what system is breaking on us? This is the part where you identify which part of the game is effectively breaking, so that you can decide easily who's job it is to fix it. Usually the guidelines will include information about what severity a bug is based on it's location. Thus, some expected baseline locations are: Graphics, Art, Sound, Input, and Gameplay. Change and add to this as you need because it's important to understand what is breaking.

Build: What version of the game broke. If you are using version control (and you better!), then it becomes very easy for the technically inclined testers and your developers to look up and understand what version of the game broke on them. This then becomes vital for knowing what possibly broke the game and tracking what build will fix the issue.

Reproduction Steps: No matter what testing you do, steps to reproduce the error are always necessary. The more involved in explaining how they occurred, the better. You should provide some example reproduction steps in your guidelines. Good reproduction steps will walk the reader through the process from starting the game to the error you are encountering.

The Guidelines

You have to have a document available for the entire team that contains templates for bug writing, the severity/location guide, an explanation for writing reproduction steps, and finally some example bug reports. In this document you can explain how bugs are handled from person to person, who is responsible for forwarding them where, and what kind of deadlines are used to ensure the bug is addressed and resolved. There is no shortage of information you can provide in this document so I'll start with the most important items, and let you improve your documents from there.

Severity/Location Guide

I put this at the top because the reason people will be referencing you document the most will be for the Severity/Location Guide and for template information. First you want to review the task tracking software you utilize for bugs and determine what kind of severity information it lets you assign. From there, give a short explanation on this note and then delve straight into the guide. 

For the guidelines themselves, I like the same general criteria that Build Breaker (our Testing Lead) came up with.

Blocking is anything that crashes the game or worse,

Critical is something that prevents play even though the application works,

Major is things that make people want to not play because they are so bad,

Minor involves more polish oriented things in which they aren't bad but aren't good either

Trivial is are things that are usually nitpicks or enhancements that you probably don't have time for.

 Knowing these as your bug types, you will often take and apply examples for every severity and location so that a tester gets a better idea of your intent. All in all, you probably won't have more than two pages of information here, use as much as you need though. 

Template Guide

Here you have the advantage of being aware what bugs pervade your game ahead of time. If you find that there are some issues that, based on location or other criteria need extra information, create a template that your testers can quickly copy and paste into other fields. Some task tracking software has templates as a feature, but I would still recommend having a record of your templates for future projects. 

Most important parts of a template! Naming conventions for bugs, empty header, description, reproduction steps, and actual vs expected outcomes. Templates are things that evolve as a development team realizes what information tends to be useful to them. A quick search for "bug report example should give you some food for thought as to what you need in your team. In any case, the report should atleast contain the information we talked about under "Bug Criteria" and in most cases, more. Things like fixed by, reported by, and description of fix are extra pieces I would recommend including within your template as they simply help get work done.

Bug Life Cycle

This is more an explanation to your team as to how bugs are handled than a quick reference. This essentially details who fills out what information, handles the bug, and ultimately closes it once it is done. In most cases, the bug is expected to be handled by a producer who then assigns it to be fixed to a developer. Then, the developer fixes the bug and attempts a back and forth with the tester to determine if the fix failed to either address the issue and/or if the fix was actually implemented. If all goes well, the bug is closed once all regressions on it have been completed.

An example diagram of the bug life cycle courtesy of www.softwaretestinghelp.com

Perform Active and Regular Tests

You need testers on your team. These testers need to not be the people making the game. Yes, they can be programmers, yes they can make fixes themselves (from time to time) but it is a good practice to have a couple testers who's sole job is to pick apart the buggy gameplay that the developers won't admit is broken. In part, you are asking your testers to be cold-hearted and ruthless in every test they do, setting the standard for the quality of game you are making.

You may cry (sorry!)

That said, many teams I've worked with/encountered are the kind of team that works only in their spare time, with few extra members. In this particular case I'd say that it's absolutely necessary that you have one of your programmers take up the mantle of SDET/Test Lead and read this (and other) articles! I cannot stress how important it is that someone have the dedicated task of arranging how and when testing happens, especially in a volunteer team with limited time. Testing needs to happen every week, for every iteration, and needs to check off on every release. 

That out of the way, I'd like to broach the topic of test cases. In a traditional testing environment, test cases take the role of instructing and reminding the tester what to check and where to check it. These generally start as a short list of things within an email, notepad, or even excel document.

Your entire goal with these test cases to understand the pitfalls of your game in an iterative sense. I'd suggest writing short test plans that address every control scheme, interaction, level, entity and state of the game that you can think of. Once you've done that, come back to the test plans on a regular basis, like after a sprint, and add any new cases to check for. The test cases themselves should be simple pass/fail affairs, though their actual state can be pass, fail, blocked, N/A, in progress, and not tested. Some examples:

Every button in the menu can be reached in a logical manner

No input breaks the menu

Enemies deal damage on moving player

Enemies deal damage on stationary character

Damage is accurate

From there you can include an adhoc section and some notes about the goals of test plans if you want, but there is a good starting point here. Make sure you leave your testers ample time to both adhoc and follow the test plan but other than that, you've at least got the beginnings ready.

I don't have much more to say about regular testing. We setup expected dates that our build is supposed to be considered "locked" till we push out more changes from what we are working on. This generally follows a weekly schedule (Friday is lock day), and uses the branching nature of version control to support it. Testers work in dev and stage branch, we merge to dev on Friday and leave it alone after that so they can work. 

An interesting side effect of how our team is structured is that our testers, Build Breaker and Palm are actually part of the Technical team. this means that they assist with pre-production as programmers and understand the designed intent for many of the systems of the game. I mention this mainly because it's abnormal, usually testers have less technical skill (that's what SDET is for) and gain that through interactions with other's to learn things such as automation scripting and software design. Not every team will have this setup.

Perform Playtests with Interested Parties

People are interested in your game. This means you should reach out to them for play testing what you are making. We can have a long discussion about how/why of NDA's and the like later. I'm currently concerned with why, from a testing perspective, we want to hold playtests. 

The easiest answer is that your testing team has limited scope. While you can find a hundred critical or blocking bugs, it only takes one to drop a project to it's knees. Think of the "Red Ring of Death" from the Xbox 360. While many people experienced it, with a small test case the issue didn't seem that bad. In fact, I know many people who never even experienced the issue. It still had a dramatic effect on the success of the Xbox 360 and can be used as an example of bugs that can't be found without more hands on the product at one point or another. 

My suggestion, is that every project should have some semi-regular play testing dates set aside that people can look forward too. Before you reach those dates though, I want you to consider all the house keeping you need to do for this. Things your play test should include:

A start and finish date

Outreach (get the people you want ahead of time)

Dedicated staff (people managing the event)

A subscription form (always collect email addresses for people to track you)

A bug reporting form

Someone to filter and report the bugs from the form

A plan

I'm not going to go in depth into the organization of a play test here. I think it's a topic that deserves it's own post, and I've spent more than enough time just on the topic of testing here. Thing is, it's important to realize that every single piece of your team wants to know what is happening during the playtest, so it's an event that you cannot be lazy about in any circumstances.

Reach the Bug Acceptance Ratio before launch

Every single thing you do in testing comes down to this last statistic. This is the point where you actually sit down and address how the game is shaping up bug wise before the intended release. So, what is a Bug Acceptance Ratio? Well, remember that severity of bug discussion we had earlier? Simply put, Bug acceptance ratio is the number of each severity of bug you are willing to allow into your final product. In many cases this comes down to something like: 0 blocking, 0 Critical, 1 Major, 10 Minor, and 100 Trivial. 

So, as the project nears it's release date, it becomes important for the developers to address the bugs almost in exclusion to other development. This may mean resolving bugs that aren't problematic as by design or will not fix, but each resolution has to be a decision that is agreed upon by tester, developer and producer. You can't simply "By design" a bug that crashes the game, you just can't. There's a special place in the bottomless pit of tester hate for people that do this sort of thing.

Really, the best way to approach this step is to realize that the production cycle of your game fundamentally changes in the last month to 6 months of development. This is the point where new content slows to a crawl, and existing content is iterated on for improvements. Take this time to think about what you've done, and improve it in every way possible. Address the nit-picky problems that have been bugging your staff and get creative to fix them. Just remember, at the end of production you have to ship and move on to the next game. 

Closing

I give a lot of ultimatums here and I would like you, the reader, to realize that testers really care about making games. Sure they come down hard on developers (like an ant telling Goliath to knock it off really) but this is strictly out of a want to see the game improve. Truth is, each and every one of us don't want to see a single game devolve into the unforgettable buggy mess that so many do. More good games always make for a better world.

Useful Resources

Interview Questions for QA Tester (http://qaquestions.net/)

Never has a better or more succinct list of what a tester does been compiled. If you want to know everything a tester should know, this is a great starting point. It even covers things considered to be SDET knowledge.

Software QA and Testing Resource Center (http://www.softwareqatest.com/index.html)

This site is a treasure trove. It contains everything from FAQ to tools, to even books that are useful. You might not find every answer you are looking for but you will find a lot of answers. Barring the god awful presentation of course.

Software Testing Help (http://www.softwaretestinghelp.com/)