Advanced Retention Metrics in Free-To-Play
I decided to put together a free eBook PDF on some of the lessons I’ve learned from free-to-play game development, from companies like Supercell and Next Games.
What is retention?
You can think about retention as a funnel. You have all the players, let’s say N amount of players at the beginning of the funnel. This starting point is when they’ve installed the game and are launching the game for the first time. They get into the game, and eventually, they close the game. They gave the game a try, but will they return, and will they continue playing? Retention metrics are meant to measure how well a game can get their players to come back to the game, tomorrow, in a week, in a month and even years from now.
We’ll now cover the most important aspects of retention metrics that game developers need to understand.
The way to make sense of retention metrics is to have some retention benchmarks from other games. With these benchmarks, you can understand if your new game is doing well or not, and how far you are from the other games that are live in the same gaming genre.
A few resources for retention metrics are GameAnalytics, SensorTower, and AppAnnie reports that they usually publish annually.
We’ll cover the benchmarks for the different retention metrics under each of the metrics listed below.
To have accuracy for retention metrics, you want to have hundreds or even thousands of players playing your game. For new retention numbers and playtimes, you can rely on a few hundred players. Once you start to look at players coming back after a week from day-0, or even a month, you’ll need over a thousand players in your cohorts.
The reason that long term retention metrics require more significant cohorts is that more people will drop off from the game every day. If you’d have a cohort of 100 players on day-0, you might only have a few players left after 30 days who are still playing. The statistical significance will not provide enough accuracy with only a few players, as it might just be that these players aren’t playing every day, and the numbers start to look weird.
But when you have 1,000 players on Day-0, you should still have dozens of players playing a month later, and the accuracy will be a lot better.
When you start looking at retention metrics, it’s crucial to look at players as cohorts. The cohorts are groups of people with specific similar characteristics. Here we break down some of the traits.
Let’s say you have 1,000 players playing your game. To look at their retention metrics, you want to look at players who started the game with the same version of the game and who came from the same country. Why do these matter?
- The version of the game should be the same because if you’ve done changes to the game and put out an update, the changes to the game will affect the player behavior. They might retain better or worse. To make sure that your data is accurate, look at players who started with the same version.
- Country-based grouping is essential, as, in some countries, the preference to play games can differ per country. Here are a few ways it can happen: A) English menus and texts won’t hurt the experience versus wanting to play localized content. B) Public holidays, free days from work can differ per country. When people have more time, they will play games more.
What can go wrong if you neglect cohort-based measurement? You might end up mixing up players, who are further in the game, to the players who just started playing. From the following metric that we will discuss, Day-0 playtime, developers might measure all their player’s playtime and use that as measurement.
Another place where developers often struggle is that they are worried about small cohorts, where they might be trickling fifty to a hundred players into the game every day. However, summing up a week’s worth of players, they can start looking at Weighted Average for the cohort of players. They need to make sure that the players came in with similar conditions, like game version and country.
Before retention starts to matter, you need to look at Day-0 session times. It’s the first indicator of how likely it is that people will return to the game. Day-0 talks about the day when the player entered the game for the first time. We measure the engagement of players, with the early phases of the game.
In 2018, Google Play launched a report on retention. The report revealed that Day-0 playtime is the highest indicator for players coming back. The more time the player spends in the game on the first day, the more likely it is that the player will go back. This trend was similar, from top-performing games to low performing ones.
Note: the report talks about Day 2 Retention, which is more commonly talked about at Day-1 retention in the games industry. But it’s talking about the same thing 😊
As you can see from the above picture, an excellent benchmark for Day-0 playtime would be to achieve 10 minutes of average playtime on day-0.
To calculate Day-0 playtime, take a cohort of players for the same version of the game and look at the summed up average time that a player would play during Day-0.
Here is a simplified example of four players. In real life, you would want to have hundreds of players to have statistical significance. Note: You’ll need fewer players for Day-0 metrics to have statistical significance because you will still have 100% of the players on Day-0.
Retention Day-1 is the cornerstone metric of free-to-play games. A player starts playing a new game, and they decide to come back to the game on the following day. When you have a large cohort of players, at least over 500 players, you can, with statistical significance, measure their Day-1 retention.
To get retention Day-1, you take a cohort, and you look at them coming back to the game, one day after they started playing.
The cohort needs to return on the exact following day from Day-0. The reason for this strict rule is simple. If you are calculating something that is called the Rolling Retention, where you’d be looking at Day-1 and including all consecutive days contributing to the metrics, you’d have a metric that becomes better with each successive day as more people come back at later dates. Strict rule: By looking at the exact following day from day-0, meaning day-1, you’ll have an accurate and final number for decision-making.
Here are some benchmarks for Day-1 in mobile games, which are pretty much the same for every gaming genre.
- Less than 30% coming back on Day-1 is lousy. You’re losing a devastating amount of players who don’t want to continue playing the game.
- 30% to 39% coming back on Day-1 is not the end of the world. If you make significant improvements to the game, you can push the metrics to improve.
- 40% to 45% is an industry-standard for a decent retention day-1 metric. If you’ve achieved this metric, you have something interesting in your hands that players will want to play.
- Above 45% is excellent, and you can move on to optimizing retention on the following days like Day-3 and Day-7.
Retention Day-3 / Day-1 ratio
Once you have convincing Day-1 numbers, at least over 40%, you can start paying attention to Day-3. Once you’re confident that players have enough content to enjoy Day-3, pull out the Day-3 retention numbers.
When you have Day-3 numbers, compare the ratio of Day-3, divided by Day-1 numbers for the same cohort. This ratio matters because it will give you guidance on how well players are staying in the game. The ratio operates similarly to the Day-0 playtime, as it’s an indicator for future retention numbers.
This table is from Traplight Entertainment, from a blog post that they share in 2019, as they were soft launching their game “Battle Legions.” You can see that they are focusing on Day-3 / Day-1 ratio. As the post says: “The focus is first on day one and day three retention (D1, D3) and what we look more than just the separate numbers is the ratio between those two: The D3/D1 curve (and later D7/D3 or D28/D7) gives us a prediction of the longevity and appeal of our game.”
Once it hits 0.7, they were happy and moved on to improving other metrics.
Further reading on retention metrics
You can get my free eBook on retention, in PDF, by going here.