Designing incrementality experiments can be tricky for businesses that have substantial amounts of natural or induced seasonality in their business.
Big events like black friday or cyber monday discounts can be make-or-break for certain businesses and so it can be critical to understand the incrementality of advertising during this time period. However, these time periods are also very much not “business as usual” and so marketing analysts need to think hard about how to design and run experiments during these critical time periods.
Generalizability
One key issue with experimentation in the context of marketing is that the results are time-bounded. That means:
- Incrementality experiments run during the summer might not apply (at all) to your business during the holiday period, and
- Incrementality experiments run during the holiday period might not apply to your business at other times
This is a “generalizability” problem which just means that we aren’t sure if the results from one specific experiment generalize well to other time periods. There’s not much that you can do about this practically other than to be aware of the problem and make sure you’re communicating the potential limitations of the experimental results.
Changing Variance
In the context of the analysis and design of experiments we care about the amount of variance in our outcome variable because that impacts the absolute amount of “noise” (or random variation) we can expect to observe and therefore impacts how much lift we need to observe in an experiment in order to observe a statistically significant result.
That is: the more variance there is in our KPI, the larger a lift we need to observe in our experiment to be confident that it’s real and not just random noise.
This matters around the holidays because you need to take this into account when you are designing your experiment. If you design your experiment for “business as usual” behavior in September, it might yield an experiment that is well-powered in September but that is not well-powered in December when purchases and variation in purchases goes way up.
To address this, you need to take the additional expected variation into account. The best way to do this is to design this year’s experiment using last holiday period’s sales data. You can do this in GeoLift by Recast by simply changing the data you upload to end at Christmas of last year (for example) so that you don’t design the test based on the intervening “business as usual” periods.
Confounding Events
The last thing to be aware of is that the holidays often come with a plethora of “confounding events” that marketing analysts need to be aware of when designing and analyzing experiments. These type of events can come in the form of a last-minute email blast that ends up hitting certain geographies more than others to try to juice the numbers before the shipping deadline, or some special holiday season “pop ups” that are being launched in select geographies.
Whatever the event, marketing analysts need to keep an eye out for these type of activities that could invalidate the results of your experiment if you’re not careful. In the case of the email blast you’re just going to want to confirm that the email isn’t targeted to specific geographic regions and instead is spread evenly. If there are “pop ups” happening you’re going to want to potentially exclude those regions from your experimental analysis (both test and control) so that you don’t accidentally end up with a mixed read.
These types of confounding events can happen any time of the year and so you always need to be careful of these when interpreting the results of your incrementality experiment. However, these types of events seem to be more common during peak buying seasons when pressures are high so you need to stay extra vigilant!
Conclusion
Running incrementality experiments during peak buying times can be an important way to get signal about what really works and to learn things that can’t be learned at any other time of the year.
However, when designing and analyzing experiments during these abnormal times you have to be careful of these common mistakes so that you can make sure you get good, valuable results at the end of your experiment.
