Should You Respond To This Metric Change?

On a daily basis, our business metrics may either increase or decrease, and occasionally, we may observe significant spikes or drops. When a change in metric occurs, we evaluate whether it warrants a response or if it is simply an anomaly that we can ignore.

Six years ago, a friend posed a challenging question to me – how to determine which changes in metrics require immediate attention? I had no answer then, and Universal Analytics was of no help since it lacked an anomaly detection tool

However, I refused to give up on this problem and spent countless hours researching and studying. Today, I possess a reliable solution that can accurately identify when a change in metrics has occurred and when to take prompt action. 

The solution has been game-changing for my clients, and I guarantee it will be just as beneficial for your business. 

Alright, let’s start and get right into it. I’ll begin with the first fundamental solution.

Compare to Previous Times Span

Comparing the present time span with previous ones is a simple yet essential solution that greatly benefits small and medium-sized businesses. This approach does not require any additional budget allocation or expertise in statistics. Doing so can provide valuable insights and help businesses make informed decisions.

Google Analytics 4 Traffic Report Comparison of October 2023 with October 2022
Google Analytics 4 Traffic Report: Comparison of October 2023 with October 2022

It is also essential to mention that you can compare the current time period with the same time span from this year, a year, or two years ago. If your business experiences seasonality, there is no way of comparing your high season with the lowest season.

Note: There are profound ways to remove seasonality from metrics, but it’s not a part of this essay. Do you want to learn them? Reply in the comments.

Most analytics solutions (Google Analytics 4, Google Data Studio, Mixpanel, Amplitude) offer this functionality by default. Nowadays, finding a tool that doesn’t offer it is rather difficult. 

Moving Average or Rolling Average

The second solution you can use to determine if there is a trend in the metric direction that will clearly tell you if you need to take any action is the moving average or rolling average.

According to Wikipedia, the moving average is a calculation to analyze data points by creating a series of averages of different selections of the full data set.

This approach is commonly used with time series data to smooth out short-term fluctuations and highlight longer-term trends or cycles. 

In order to calculate it, just add up a set of recent data points and divide the sum by the number of time periods. 

The rolling average is often confused with the moving average. While the rolling average is the average of all values available, the moving average is the average of the last set number of values.

In the table below, I demonstrate how to calculate these two metrics in Google Sheets or Microsoft Excel.

GA4 Date

Date

Metric Value

7D Moving Average

7D Rolling Average

20230531

2023-05-31

2585

2585.0

2585.0

20230601

2023-06-01

2410

2497.5

2497.5

20230602

2023-06-02

2108

2367.7

2367.7

20230603

2023-06-03

1613

2179.0

2179.0

20230604

2023-06-04

1415

2026.2

2026.2

20230605

2023-06-05

2486

2102.8

2102.8

20230606

2023-06-06

2501

2159.7

2159.7

20230607

2023-06-07

2347

2125.7

2183.1

20230608

2023-06-08

2324

2113.4

2198.8

20230609

2023-06-09

2184

2124.3

2197.3

20230610

2023-06-10

1451

2101.1

2129.5

One of the advantages of using a moving average is that it helps to identify the exact time when a change started to occur. This feature enables you to revert to the original solution mentioned in this article, compare two different time periods, and identify the root cause that led to the change in direction.

Some analytics tools offer moving or rolling averages features, while others don’t. For instance, Mixpanel, Tableau, and Google Looker Studio offer it, while Google Analytics 4 doesn’t have it. 

Normal Distribution: Anomaly Detection

The third solution uses statistics to detect anomalies, a proven and practical approach. By leveraging normal distribution, it’s possible to find subtle irregularities that may otherwise go undetected. This method is a powerful tool for identifying patterns and outliers that can provide valuable insights and drive impactful decisions.

In time series and other statistical analyses, the normal distribution is the most commonly assumed type of distribution. The standard normal distribution has two parameters – the mean and the standard deviation.

Normal Distribution Bell Curve
Normal Distribution Bell Curve
Source: National Library of Medicine

Calculating the mean and the standard deviation for the metric period allows us to determine whether the spike or drop in the metric was an anomaly. 

To determine the anomaly, you should do the following:

  • Calculate the mean or average of the metric values for the observed period
  • Calculate the standard deviation of the metric values for the observed period
  • Calculate a 95% confidence interval (boundaries) to determine outliers (anomalies) by the following formulas:
    • First boundary: Mean – 2 * standard deviation
    • Second boundary: Mean + 2 * standard deviation

After doing the steps above, you will get two additional chart lines that will allow you to see clearly if the metric value was an anomaly or not. 

Some analytics tools offer anomaly detection features, while others don’t. For instance, Google Analytics 4 and Tableau offer it, while Mixpanel doesn’t have it.

Other options

Although there are other more advanced techniques to detect anomalies, they require the knowledge and experience of using statistics.

A few of them are STL decomposition, auto-regressive integrated moving average (ARIMA), classification and regression trees (CART), detection using forecasting, and others. 

STL-decomposition Anomaly Detection
STL-decomposition

Most of these solutions can be done in Python or R and couldn’t be achieved in Google Sheets or Microsoft Excel. Therefore, this article doesn’t cover them.

Takeaway

Managing a business involves tracking various metrics that fluctuate on a daily basis. It can be challenging to decide when to act upon these fluctuations and when to focus on other business tasks. 

This article provides a solution for SMBs to identify outliers or anomalies in their time-series data within 30 minutes and take appropriate actions. 

If you know of any other ways that SMBs can detect anomalies in their data that are not covered in this article, please feel free to share your thoughts in the comments section below.

Leave a comment

Your email address will not be published. Required fields are marked *