In marketing, every forecast comes with some level of uncertainty – whether you’re predicting sales growth, estimating the impact of a new campaign, or adjusting your media mix.
There’s no crystal ball and, too often, marketers fall into the trap of treating forecasts as absolute, when in fact, they represent a range of possible outcomes.
In the marketing world, we tend to focus on single-point estimates and struggle to incorporate probability distributions into our decision-making processes.
This article covers how to understand and navigate uncertainty better, especially around MMM.
1. The Importance of Uncertainty in Forecasts
Good forecasts should come with uncertainty. This might seem counterintuitive to those who want definitive answers, but predictions that come without uncertainty estimates are inherently unreliable.
Consider weather forecasts. It’s more useful to know there’s a 40% chance of rain tomorrow than a definitive yes or no. The forecast may frustrate you if you’re planning a beach trip, but, over time, you can evaluate whether it rains 40% of the time when such a forecast is given.
This idea of testing your forecasts helps decision-makers set realistic expectations and evaluate long-term accuracy.
Similarly, in marketing, it’s important to communicate uncertainty in forecasts clearly.
At Recast, forecasts include credible intervals or ranges that account for possible outcomes.
If the model predicts a 10% uplift in sales, it might also estimate that the actual uplift could be anywhere between 5% and 15%. This range is critical to understanding how certain—or uncertain—the forecast is.
2. Ignore Uncertainty At Your Own Risk
Ignoring uncertainty is especially prevalent in A/B testing, where companies often focus on the mean result without considering the range of possible outcomes.
For example, imagine running an experiment that shows a 5% lift in sales with statistical significance. Excited by the result, you bake that 5% lift into your next-quarter forecast, expecting a noticeable bump in performance.
But what happens if that lift doesn’t materialize?
In reality, the 5% lift could have been part of a much wider range, such as 1% to 9%.
You should always be looking at the range of the uncertainty interval and only checking the mean last. This will help you approach forecasts more conservatively – if the estimated uplift from a campaign ranges between 1% and 10%, you need to understand that the lower end is in fact a possibility and have a plan for it.
3. Handling Incrementality Testing and ROI Calculations
Uncertainty also plays a crucial role in incrementality testing. In these tests, businesses are often thrilled to report results like a 3% lift in performance, but they frequently overlook a critical component: how much was spent to achieve that lift.
Incrementality studies need to go beyond lift percentages and include return on investment (ROI) or cost per acquisition (CPA) numbers, along with uncertainty intervals.
For example, an incrementality test might show an incremental CPA of $75, but the uncertainty intervals might range from $25 to $750. This range is crucial because it tells you how reliable the estimate is.
4. Why Marketers Should Demand Uncertainty Estimates
Every marketer should ask their MMM vendor if they can easily get uncertainty estimates in the form of confidence or credible intervals for the parameters estimated in the model. If the answer is no, that’s a red flag. Good models should communicate uncertainty, and any vendor that avoids providing uncertainty intervals is likely oversimplifying their model’s results.
Recast addresses this challenge by always presenting uncertainty estimates and credible intervals front-and-center to help marketers understand the full range of possible outcomes. We want to encourage decision-makers to account for uncertainty even if it isn’t intuitive, so we present the intervals alongside point estimates.
Media mix modeling, when done correctly, provides not only answers but also questions. If the model reveals a high degree of uncertainty about the impact of a particular channel, that’s a signal to investigate further, run experiments, or gather more data.
Uncertainty should be a tool, not a limitation.
