Anomaly Detection in Gaming
By Ron Shiff
In this blog post, we will provide an introduction to anomaly detection within the context of online gaming.
We will present some examples of real anomalies that have occurred in our games and will discuss the challenges involved in implementing an effective anomaly detection system.
Game development is an exciting journey that involves creating captivating experiences for players while ensuring financial success. How can studios achieve both of these objectives? The answer lies in game analytics – a powerful tool that offers insights from game data, driving engagement, personalization, and smarter monetization strategies. However, there’s more to this story! Let’s delve into the world of anomaly detection, a superhero in the realm of data analysis for online gaming.
Imagine you’re playing a game, and suddenly something feels off – perhaps your character starts glitching or the game environment behaves oddly. These unexpected deviations are what we refer to as anomalies. In the world of data, anomalies are like red flags, signaling that something unusual is happening. In the context of online gaming, they can range from technical glitches to operational errors.
For example, in one of our games, some players were unable to log in due to a technical issue. Using our anomaly detection system, we were able to detect this issue and inform the studio.
Here’s how the graph of hourly users looked (the x-axis represents time, and the y-axis represents the number of users):
Notice two interesting aspects of this graph:
- The signal is very periodic – this is typical behavior for large segments.
- The user count values were not “extreme” (but using historical context, we were able to detect this anomaly).
Another anomaly was caused by a mistake in feature configuration by the operations team.
A feature was opened to a much larger population than intended, causing an unusual influx of coins (the x-axis represents time, and the y-axis represents the coins inflow):
This represents a completely different signal shape. In this case, since the feature is not always active, the coins inflow sometimes drops to zero.
Moreover, the signal is highly non-stationary, so we cannot rely on any seasonal behavior in this instance.
After the studios were alerted about these anomalies, they were able to quickly address the issues and minimize their impact on the players’ experience on on studios’ KPI’s.
Challenges
Anomaly detection is a valuable ally for various game development teams. Consfier studio economists, software developers and operations teams who constantly create new features to keep players engaged and happy.
By using an anomaly detection system, they can track key performance indicators (KPIs), such as player churn, resource inflows, active users, and win rates, to understand the pulse of the game. Then there are the tech wizards – the site reliability engineers – who monitor behind-the-scenes metrics like speed, hardware usage, and errors to ensure players enjoy a seamless experience. Even marketing maestros can benefit, watching metrics like user acquisition and the revenue from in-app purchases.
Studios are collecting an ocean of game data. However, the real magic lies not just in gathering data, but in deciphering it. This is where anomaly detection swoops in. It acts like a guardian angel watching over the game’s data streams, ready to sound an alarm if anything odd occurs. This vigilance is crucial because acting swiftly upon anomalies can mean the difference between success and failure.
But, just like any superhero, anomaly detection faces its challenges:
Players Segmentation
In mobile gaming, understanding “normal” behavior is complex because each player’s experience varies. The challenge lies in categorizing players into groups that reflect their engagement and spending, a task made tough by the vast amount of user data. Identifying meaningful segments that speak the business language is crucial for targeted anomaly detection. However, there’s a catch: having too many small segments can lead to false anomalies due to data volatility. Balancing the number of segments to avoid overwhelming the analysis with noise, while still capturing valuable insights, remains a significant hurdle.
Distinguishing Outliers from Anomalies
Figuring out when a player’s behavior is just unusual or a sign of something bigger can be tricky. For instance, if a few players are spending way more than others, it’s different from noticing that players in levels 100 to 500 are all acting strangely. The tricky part is removing these one-off outliers without getting rid of the real issues, the anomalies we’re actually looking for. We need to clean the data carefully to make sure we’re not throwing away important information with the outliers.
Another big challenge is dealing with outliers in real-time. Unlike a one-time cleanup of a dataset, mobile gaming data requires us to remove outliers continuously, every few minutes. This process must be super consistent over time. If we’re not consistent, even our way of removing outliers can make normal data look strange, creating false alarms. This means we have to be on our toes, making sure our outlier removal is just as steady today as it was yesterday, or last week, to keep our data analysis accurate and reliable.
“One Fits All” Anomaly Detection
Creating a single anomaly detection system that works across all segments and KPIs in mobile gaming poses a significant challenge. Most existing algorithms and studies focus on specific types of data or anomalies, limiting their applicability. However, for a mobile gaming company, the anomaly detection solution must be highly adaptable, capable of handling diverse data distributions and patterns from various player segments and game metrics.
This requirement for a generic yet flexible system complicates the design process, as it needs to accurately identify anomalies in vastly different contexts – from spending behaviors to engagement levels across different game stages. The solution must be sophisticated enough to adjust to these varying scenarios, ensuring consistent and reliable detection of genuine anomalies without being misled by normal fluctuations in player activity. Achieving this level of adaptability and precision is a key obstacle in developing an effective, one-size-fits-all anomaly detection strategy for the mobile gaming industry.
Detecting Point and Group Anomalies
In mobile gaming, anomaly detection needs to be adept at catching two main types of anomalies: “point anomalies” and “group anomalies.” Point anomalies are the industry’s usual suspects, where a single data point sticks out from the rest, signaling something unusual at a specific moment. For instance, a sudden spike in in-game purchases by a player might be a point anomaly.
On the other hand, group anomalies represent patterns of behavior over a series of data points that, only when viewed together, appear abnormal. An example could be a gradually slowing trend in daily active users, which might not be obvious when looking at individual data points but becomes clear when observing the group of points over time.
The challenge lies in the algorithm’s ability to flexibly adjust to detecting both types of anomalies without prior knowledge of the group size for group anomalies. This means the detection system must be sophisticated enough to dynamically assess data over varying time windows, recognizing when a collection of seemingly normal behaviors collectively constitutes an anomaly. Balancing sensitivity to detect anomalies accurately, while avoiding false positives in a constantly evolving data landscape, is a critical hurdle in developing an effective anomaly detection algorithm for mobile gaming analytics.
Scaling Anomaly Detection in Real-Time
The technological challenge of scaling anomaly detection to keep pace with the expansive ecosystem of mobile gaming is immense. With thousands of player segments, dozens of KPIs, and numerous mobile games, the task balloons to monitoring millions of signals. This isn’t just about keeping an eye on a vast array of data points; it’s about doing so in real-time.
The sheer volume of data generated every moment by players around the globe means that any anomaly detection system must not only be incredibly efficient but also capable of processing and analyzing data on the fly. This real-time requirement ensures that any potential issues can be identified and addressed promptly, but it also significantly ramps up the complexity and computational demands of the system.
Achieving this level of scalability while maintaining the precision and accuracy needed for effective anomaly detection is a formidable technological hurdle. It requires a robust infrastructure capable of handling massive data streams, along with sophisticated algorithms that can swiftly sift through this information to pinpoint anomalies. Balancing these demands to ensure seamless, uninterrupted gaming experiences for players, while also safeguarding the integrity of the game’s data, represents a critical challenge for developers and data scientists in the mobile gaming industry.
Summarizing and Explaining Anomalies
After detecting anomalies across multiple segments in mobile gaming, the next big hurdle is effectively communicating these findings. We’re often left with dozens of segments showing anomalies, and the challenge is to distill this complex information into a clear, concise summary that can be easily understood by customers or business stakeholders. The summary not only needs to be accessible but also speak the business language, translating the technical details of anomalies into actionable insights.
Moreover, beyond just identifying anomalies, there’s a pressing need to provide explanations or potential root causes for these irregularities. This could involve integrating and analyzing live-ops data, player behavior trends, or game update logs to offer a comprehensive view. The goal is to not just highlight the ‘what’ but also to shed light on the ‘why’ behind the anomaly. Achieving this requires a deep understanding of both the game’s mechanics and its business implications, ensuring that the summaries not only inform but also guide decision-making processes effectively. This balancing act of precision, clarity, and insightfulness in communication is crucial for driving informed actions based on anomaly detection findings.
Summary
We discussed the vital role of anomaly detection in the world of game development, and how it acts as a guardian angel for game developers, identifying unusual events or issues that can impact player experiences.
We presented some the of challenges in applying an anomaly detection system in online games and mentioned some of the anomalies we identified with our system.
In our next blog posts, we will explain how we tackled these challenges and provide some tips on how to address them.
