#telemetry data

It depends on how the survey data is collected. If it's collected from in-game or from a place that you're logged in via your game account, then yes - we have total knowledge of who you are (minus any information necessitated by legal privacy requirements), how much you've spent, what you've spent on, what region you play in, how many hours you spend playing, and so on.

If it is from a normal web site that doesn't require you to log into your account, it is sometimes generally treated as if it were mostly anonymous (the region you're browsing from, for example, is still still visible via IP lookup). If you have any tracking cookies in your browser history, it is possible for the site to gather more identifying information about you as well. Some big tech companies also do some pretty sneaky stuff with their website code - [LinkedIn was recently discovered to scan (quietly) for 6000+ specific browser extensions whenever you visit the website in a massive data scraping measure].

This information is aggregated and cross-referenced for metadata - we might not know exactly whether respondent A spent money or not, but we do know that respondent A is from the USA/Canada region in Central Standard Time, and that roughly X% of players from that region spend $Y every month and so forth. This information gathering generally isn't to find out about individual people (unless those individuals are particularly notable like the extreme whales whose individual spend is a significant portion of our total revenue), but to identify trends and overall sentiment across the player base. We need to know who likes what (and how many of them there are) so that we can make sure that the most players get at least some of what they want us to make.

[Join us on Discord] and/or [Support us on Patreon]

Got a burning question you want answered?

Short questions: Ask a Game Dev on Twitter

Short questions: Ask a Game Dev on BlueSky

Long questions: Ask a Game Dev on Tumblr

We mostly track all the things that players choose to interact with. This means where they go, the things they spend time looking at, the NPCs they talk to, the things they buy, the recipes they craft, the menus they spend time in, the characters they romance, the gear they like to use, and so on and so forth. You know how achievements often require a bunch of very seemingly-arbitrary conditions to meet, like visiting Calendar Man at several major holidays in one of the Batman Arkham games? A good rule of thumb is that if we can check for it for an achievement, we're probably tracking it. We spend a huge amount of time and effort tracking all that information because we want to know what the players are spending their time doing in order to figure out what kind of new content we think they will like.

I think most players don't really understand how much tracking we do for UI/UX. We spend a lot of time and effort to track things like how many players don't continue beyond a certain point in order to figure out how we can make things smoother and easier for players to get to the fun bits. A huge amount of effort is spent tracking, analyzing, and interpreting data from the First Time User Experience (aka the FTUE), because we all want players to keep playing and not drop the game after the first few minutes.

[Join us on Discord] and/or [Support us on Patreon]

Got a burning question you want answered?

Short questions: Ask a Game Dev on Twitter

Long questions: Ask a Game Dev on Tumblr

Whenever a player leaves a 5v5 game mid-match, they leave their team significantly weaker than their opponents. Team fights are lopsided, resource management like gold/experience/jungling is lopsided, collective actions per minute are lopsided, and so on. Uneven games are almost never a good player experience - the longer a game with uneven teams goes on, the more lopsided the game becomes.

Backfilling the player slot mid-match doesn't fix everything either - the replacement player doesn't necessarily know how far into the game things are already, the character they choose may not go well with the team, and there's also the question of gold/experience/items to distribute to that player. Unbalanced games are not a situation the developers wish to encourage.

Obviously, leaving a game due to circumstances outside a player's control can be excused. Things happen - the internet goes out, a player has some local emergency, and so on. However, we developers wish to discourage player habits of repeatedly and/or intentionally dropping from games because of how the act negatively affects the experience of the other nine players involved. This generally means that players who are banned for such behavior have a history of doing so.

The games we build today have automated data collection on what occurs in the game. We have chat logs of who said what when, we have input logs of who clicked what when, we have the entire game replay that we can review if need be. We keep records of all of that data for each game that players play and can call the game data back up when we need to. This way, we can review the full game history of the players we investigate as needed, especially if they've been reported by other players for other bad behavior. Large games like League of Legends even use machine learning on its recorded game data to identify and review bad behavior cases.

Ultimately, the devs' goal is to encourage good behavior and discourage bad behavior among players. We don't actually want to dole out punishments if we don't have to, because punishing bad actors isn't the point. What we want from bad actors is twofold - we want to reform the bad actors who can be redeemed so that their bad behavior stops and, if we can't reform them for whatever reason, we ban them. The players who receive permanent bans are typically those who will not stop their bad behavior.

[Join us on Discord] and/or [Support us on Patreon]

The FANTa Project is being rebooted. [What is the FANTa project?]

Got a burning question you want answered?

Short questions: Ask a Game Dev on Twitter

Long questions: Ask a Game Dev on Tumblr

Today I’ve got a war story from one of the developers on our [Discord server]. It’s an epic saga that encompasses years of development experience and investigation. I hope you like it as much as I did.

By now, it is fairly common knowledge that video games collect a tremendous amount of user data but, aside from financial reports, it is still fairly opaque about how those data are used. The following is a true story about how data made a tangible change in a live game.

When I started my job, I was asked to play through several hundred levels of our flagship title, a mobile puzzle game, as though I were playing for the first time. I wrote down thoughts and feedback about how I felt playing the game, while learning about how we made content. As I was playing, I got stuck on one specific level early on - Level X. It took me 44 tries to beat it. It was very precise, requiring the player to break specific pieces at specific locations in order to win. If I were a real user, this experience would have made me quit the game. I brought this up at the end of the week, and my lead revealed that "Level X is the best performing level in the game." No other level converted non-paying players to paying users as well as Level X. It singlehandedly caused the most purchases and generated the most revenue out of hundreds of levels. This shocked me. What I played caused only frustration - how could this level be our best? 

In addition to my normal work, I started asking about what data we gathered and why. Data informed how we created levels and designers were encouraged to understand it. I learned that data fell into two large buckets: Big-picture, business information to determine the health of the product and small-scale level-by-level data points. The first bucket includes concepts like revenue, retention (the percentage of people who keep playing at key time intervals - 7 or 28 days after their install date), conversion rate (percentage of free users who become spenders), daily active users, number of new installs. At the time, our second bucket full of small-scale metrics only included a few data points: Play Count, Win Rate, Quit Rate (% of plays that end before you run out of moves), and Monetization (revenue, in-game spend, power-up usage, and conversion rate). Level design primarily focused on win rate and level monetization.

I learned that Level X had a low win-rate of 4%, outside our normal difficulty target for the “hardest” levels at 5-10%. However, my lead was absolutely right - no other level came remotely close to Level X’s monetization metrics. Looking at the data that we had, everything shown indicated its low win rate was worth the revenue generated. However, something about looking at the levels around it puzzled me, because I could see far fewer play attempts on levels after Level X proportional to their difficulty. It seemed like we had an incomplete picture. Something about Level X didn’t sit right with me so I decided to investigate further. What followed was a multi-year process of revising how we gathered and analyzed data, and the most complicated spreadsheet I've ever worked on. 

The core of this effort focused on Churn Rate. Churn is defined as "the number of customers who leave your product within a given time frame." It took me, and another designer, and a product manager six months of working part-time to revise our metric tracking and validate it through live testing. We learned that Level X had a churn rate of an incredible 15%, double the rate of every other level in the game, and 4x the average of a level at its win-rate. When we presented our findings, the entire team sat in stunned silence for a few minutes - Level X defined an era of level design. Everyone strived to make a level that "performed as well as Level X." Were all of our premises about level creation misguided? What do we do now?

Come back next time for the conclusion to the tale - did they actually fix Level X? How could they do that?

[Click here for the conclusion]

[Join us on Discord] and/or [Support us on Patreon]

The FANTa Project is being rebooted. [What is the FANTa project?]

Got a burning question you want answered?

Short questions: Ask a Game Dev on Twitter

Long questions: Ask a Game Dev on Tumblr

Today we conclude the guest tale that we began yesterday. You can read the first part by [clicking here], but here’s a quick summary - we’re at a mobile game studio whose flagship puzzle game has a legendary Level X that historically earned the most revenue of all of their levels... but was also so frustratingly difficult that 15% of all players who played it were so frustrated they quit and never returned. What to do?

The answer was “we try everything”. We didn't know how broad our changes had to be. Could we replace just the level itself? The levels around it? The entire early content path? How much time could we devote to this, while we still have a live game to run and update? Every designer on the project created multiple variations for Level X. We ran a sweepstakes for who could make the best-testing level. A splinter team started updating the rest of the first hundred levels. We ran dozens of A/B tests, where representative groups of players receive different content than a control group. Some were negative, some were inconclusive, some pointed us in a positive direction. We went through a full year of testing, and we'd finally come to a best-of test. It was our riskiest yet because we were changing the entire tutorial and difficulty pacing for the first 50 levels of the game - the entire new user experience.

For a month, the entire team was tense. The worst part about pushing test changes live is the waiting. Changing content requires getting new players to install and run your game, getting put into a cohort, and actively playing for weeks. Acting on early data could lead to huge repercussions. It influences revenue, retention, player count, player happiness - the health of your game as a whole. 

After that first month, we all breathed a collective sigh of relief. Our test cohort was leading in every one of the metrics. Level X was entirely different and, while it was still difficult and generated a third less spend, its churn rate was cut to 3%. The effect this had on levels after Level X was phenomenal. The block of the next 50 levels were seeing much larger amounts of players, increased spend, increased revenue. Our day 3 and 7 retention numbers were up by the highest we had ever seen in a single test. We pulled the metaphorical lever to release the content to every player.

Two years after I joined the team, Level X was replaced and the game was in a much healthier place. Our team's mindset changed to "If you build a game people keep playing, people will pay for it." Our design philosophy changed and we no longer made "pinch point" levels like the old Level X. However, no test is perfect, no content without flaws. In replacing levels, we made some better, some worse. We celebrate our wins, learn from our mistakes, and move on to our next test.

I want to thank my guest poster for sharing their war story with us. Our ability to gather feedback and refine our game design has evolved alongside the technology and online connectivity. Most dev studios now have analytics teams that track this kind of data and use it for both business and game design purposes. It’s valuable to see how players interact with the things we’ve designed because it can (eventually) lead to better player experiences overall.

[Join us on Discord] and/or [Support us on Patreon]

The FANTa Project is being rebooted. [What is the FANTa project?]

Got a burning question you want answered?

Short questions: Ask a Game Dev on Twitter

Long questions: Ask a Game Dev on Tumblr

We get access to data directly related to our products but not others at the discretion of the platform. So, for example, my company wouldn’t get use data for PS4 Spotify users from Sony, but Spotify could. It’s also important to recognize that there’s stuff that’s tracked by the platform and there’s stuff tracked by us. The platform controls what we get to see (some of them are more closed than others), but we get all of the data from our own stuff.

Generally speaking, third party devs like me can track any data that directly affects the game itself. This includes anything that you can see or do or access in game, the quests you take and complete, the gear you like to use, the number of characters you create, which alien you decided to romance, how much time you spent in game, and so on and so forth. This also includes any DLC or additional content that you download and apply to the game and any trophies/achievements you get while playing the game. The game’s control mapping and in-game resolution also directly affect the game, so we can see and track those too. If it affects the game in any way, we have to be able to see it because we have to be able to make the game respond to it.

[Join us on Discord]

The FANTa Project is currently on hiatus while I am crunching at work too busy.

[What is the FANTa project?] [Git the FANTa Project]

Got a burning question you want answered?

Short questions: Ask a Game Dev on Twitter

Long questions: Ask a Game Dev on Tumblr

Sure thing. First the how: 

[here’s a link to the basics on what kind of telemetry data we gather]. 

If you want a deeper dive, try the [#telemetry-data tag] on this blog. 

Now the why.

Since games have existed, game developers have needed feedback in order to improve. The game can’t get better, more engaging, or more fun unless we get feedback on what we did right and what we did wrong. So we’ve been gathering information about what kind of games you like, what features in games you like, what sort of things you’d be willing to spend money on, what you dislike, what causes the most frustration, what you want more of, what you want less of, and so on and so forth. We’ve done this through many different means - surveys, focus groups, playtests, interviews, trade shows, word of mouth, reading feedback posts from social media and web forums, etc. 

Getting feedback is hard. Getting honest feedback is even harder. Getting specific feedback is really hard. Non-dev playtesters often lack the right vocabulary or context to describe their feelings adequately about the game after a playtest. They just don’t have the right context to put it into words, especially if they aren’t core gamers, and AAA+ games always aim for the broadest audience possible. Playtesters also tend to try to answer the way they believe they should feel rather than how they actually feel, especially in groups where they feel they’ll be judged by their peers. And it’s also super difficult to get them to remember specifics about the play session unless we really keep the play session laser-focused on just one thing. If a playtester plays a new game for twenty minutes, chances are good that they will have to pick up at least 3 or 4 different game systems, often many more than that. If we ask about something in specific, we often either get vague answers or nothing at all unless it was one of the major game systems (like combat) that they engaged with. And, of course, there’s no way that we can pull in millions of players for playtests without automation. We don’t have that kind of time or budget.

What does this have to do with monitoring players?

Telemetry data allows us to track what players do during play sessions, and that information honestly tells us what they were doing in the game. The data doesn’t tell us what we want to hear, it doesn’t try to make itself seem like a more hard core player, and it doesn’t ever forget the specifics that we look for.  That information is invaluable in telling us where players had the most trouble and the things they spend the most time doing. We don’t base all decisions solely on telemetry data, but it provides valuable missing perspective when we get other forms of player feedback that we can corroborate. It provides us a more accurate picture of the player experience so we can identify the problems that need fixing and the things that players enjoy most. We need that information in order to improve games. 

When you see things like quality of life improvements, more intuitive interfaces, a better tutorial process, more engaging character arcs, smoother quest flow, better dialogue, and overall hard-to-quantify gameplay improvements, you’re actually seeing the results of professional developers taking feedback from players at scale and incorporating it into their work. Much of that feedback and knowledge comes from gathering that telemetry data, because honest, accurate, and specific feedback is super duper hard to get otherwise. So… no, we probably aren’t going back to the age of unwatched players. That would make the gameplay experience much, much worse for everybody involved.

The FANTa Project is currently on hiatus while I am crunching at work.

[What is the FANTa project?] [Git the FANTa Project]

Got a burning question you want answered?

Short questions: Ask a Game Dev on Twitter

Long questions: Ask a Game Dev on Tumblr

No publisher expects a game’s audience to grow, or even stay constant forever. Every live service game has a live team that constantly monitors the numbers and player behavior. This doesn’t just include customer service handling player tickets, but also data scientists who track, parse, and interpret lots and lots of data. We really track a ton of it.

One of the most important interpretations we have of data is tracking trends. By tracking the number of active players on a day to day basis vs a week to week and a month-to-month basis, we can get a better view of the total community behavior overall. We can see how much time the players are playing on average, their aggregate spending behavior, and the breakdown of time spent questing compared to pvp, raiding, crafting, single-player content, standing still, chatting, etc. For a more in-depth look at these data, [click here]. If you really want a deep dive down the rabbit hole, try the [#telemetry-data] tag.

What all this data analysis does is enable us not only to look at what came in the past, but make predictions for the future based on extrapolations and trends that we observe. Despite what many players want to believe, we actually want the most accurate numbers we can get so that we can make the most realistic predictions possible. These predictions establish our expectations going forward. Every game service expects the player population to fall over time. People get bored, they find another game, real life situations change, or whatever. We expect attrition. That’s the nature of the beast. The question is how much attrition do we expect, and whether the predictions were accurate.

If we can predict the rate at which we lose players, we can plan our budget spend accordingly to maintain profitability even while the population dwindles. At some inevitable point, the number of players will be insufficient to justify further development spending, and that’s when support for the game will end. But until that point, the remaining players continue to receive updates and new content, commensurate with how much we can earn from developing it.

The situations where it gets worrisome is one of two possibly-overlapping situations: 

When the predictions are very wrong

When the rate at which we’re losing players is way too high

If our predictions are super wrong, it means that any decisions made to determine future development budget are probably wrong and we need to figure out what’s what. If the rate at which we’re losing players is far too high, then the publisher may yank support for the game service altogether because it isn’t financially sustainable. Ultimately, be it Fortnite, PUBG, League, WoW, or Rainbow Six Siege, the methodology is the same. Use the data to build a predictive model in order to figure out what we think will happen, and use those predictions to determine future development budget and scope.

The FANTa Project is currently on hiatus while I am crunching at work.

[What is the FANTa project?] [Git the FANTa Project]

Got a burning question you want answered?

Short questions: Ask a Game Dev on Twitter

Long questions: Ask a Game Dev on Tumblr