#statistic modelling

This article about negative uplifts belongs to a special series of blogposts, written by our own data wizards. It will offer you a glimpse into the engine room of Mediasynced. In these informative blogposts, we shed a light on the complexity of TV performance measurement in realtime and our robust statistical solutions.

One of the primary ways to evaluate the effect of your commercial is by comparing the performance before and after airing your commercial. A baseline is calculated based upon the performance before the commercial and is then subtracted from the performance after airing. The difference is then called the uplift. This process can be done for all kinds of metrics: we can calculate the uplift in the number of sessions on your website, the uplift in conversions and many more.

While this method can give you a good estimation of how well your commercial performed, it is not perfect. One of the most obvious flaws is that it can result in negative uplifts: cases where the performance after airing a commercial is worse than before. It is quite unlikely that users will start boycotting your products after seeing your commercial, so what is going on here? Is it even possible for a commercial to have a negative effect?

What is actually happening is that the commercial did not have a noticeable effect and the results you are seeing are actually due to noise. There is no reason to be alarmed, you did not actually lose any potential customers due to your commercial.

In the example shown above the goal is to calculate the uplift in number of sessions on your website after airing a commercial. The number of sessions is not static, it changes throughout the day in an unpredictable manner. In the graph we can see that right before the commercial aired, the number of sessions started dropping. The effect of the commercial was not enough to compensate for this drop, resulting in your actual performance being lower than the calculated baseline.

So now that we know that these negative numbers are not representative for what is actually happening, what can we do to improve them? As discussed above, the problem is that your commercial did not have a significant effect. One method that some of our competitors use, is to set all negative uplifts to 0. While this sounds like a good solution, it is statistically unsound. There are also cases where the exact opposite is happening such as the example shown below. 

The effect of your commercial is also insignificant in this example. The number of sessions started increasing right after the commercial was aired. This is not an effect of the commercial, but caused by random fluctuation in visitors. However, this situation will result in a large uplift. If you set all negative uplifts to 0, then you must do the same for the positive-uplift case (which is much harder to identify) where the effect of the commercial was insignificant, yet the uplift was still large. If this is not done your results will be positively skewed. Your overall results will be much more positive then what is actually the case. Often this results in surprisingly large uplifts for small channels, as these channels are much more likely to have small effects which may result in incorrect positive and negative results. The negative results are set to 0 so you are left with an average uplift for this channel much higher than what is actually the case. When you leave these negative numbers as they are, incorrect positive and negative results will cancel each other out over a campaign period. This results in a much more accurate estimation for a channel.

So if we can’t ignore these numbers then what can we do? The statistical reality is that we simply can’t know for certain whether a session or conversion is caused by your commercial or has a different reason. As this problem is a fundamental one in all of statistics and is not only limited to the problems in our branch.

What can be done is limiting the effect of this problem by employing methods that filter the real data from these noise and predictive models that can make strong estimations of the real data.

Mediasynced is constantly improving its methods and models using the statistical and Artificial Intelligence tools that we have developed to provide an accurate prediction. Such an example is the method we use to calculate the baseline. You can read about this method here.

This article about the baseline belongs to a special series of blogposts, written by our own data wizards. It will offer you a glimpse into the engine room of Mediasynced. In these informative blogposts, we shed a light on the complexity of TV performance measurement in realtime and our robust statistical solutions. Hopefully you will enjoy reading the very first blogpost of this special series. 

The common way to measure the performance of TV spots, is to measure your website in the minutes following a commercial and comparing these results to the performance that we expected without a commercial. This process is called benchmarking. Your first step would be to measure the amount of sessions your website has after the commercial. The next step is to create a benchmark, also called a baseline to compare your results with.

Linear Baseline

The simplest method would be to measure the amount of sessions on your website before you aired the commercial and take that number as the baseline. While this solution may sound intuitive and adequate in theory, in reality it is just not accurate enough. The first problem is that your benchmark is static. You assume that it won't change during the period after your commercial up to the point you stop measuring.

To fix this problem you can use two measure points instead of one. In addition to the measure point before airing the commercial you now also measure the point after the effect of the commercial has faded. Now you can simply draw a line from the first measure point to the second and use this line as your benchmark. The problem here is identifying when the effect of your commercial has faded. We will discuss this problem in a future blog.

Another problem is that this benchmark is based on a short snapshot of your website’s traffic. In such a short timeframe, sudden variations in your website visits can occur that will distort the baseline calculation. So the measured baseline may no longer be an accurate representation. This can result in significant higher or lower measured uplifts than what is really the case. So adding an additional measure point does not change anything. Now your baseline is simply based upon two potentially inaccurate points.

Average baseline 

So why don’t we take the average traffic of each specific day and minute combination? Well that model is flawed because there is no such thing as an average Monday. Seasons, the weather, events, etc., can all heavily influence the baseline.

Our Baseline

At Mediasynced, we have taken a more sophisticated approach in order to provide an accurate baseline. For each client, we build specific day models. These models are based on historical data and take several factors into consideration. They are then aligned with the actual data with the use of a y-intercept. Using this method we can construct a baseline that follows the natural curve we expect it to have, so based on the organic rhythm of the website throughout the day. If we expect the traffic of your website to rise slightly from 16:10 to 16:20, then the constructed baseline for that time frame will also rise slightly. To increase the accuracy we have splitted our data up by device and repeated this process for each of those devices. Using these methods we were able to construct a baseline that correlates better with reality and is less affected by the noise within the data.

In the example below, we see that the number of baseline sessions (real baseline) is oscillating around an average of 10 sessions per minute. During the commercial the baseline is starting a positive peak which the two simple baselines fail to capture. Our baseline on the other hand can correctly predict this pattern and recreates this peak as best as possible.