More and more brands are running lift tests. That’s a good thing.
Incrementality experimentation methods are not that complicated — and you don’t want them to be. The value of a structured experiment is that interpreting the results should be straightforward.
But in practice, we see some common ways teams misread lift test results, which can lead to bad forecasts and wasted budget.
Here’s what we see often goes wrong, and how to fix it:
1. Confusing Statistical Significance With Certainty
Most marketers first learn how to interpret experiments in the context of A/B tests, either for product features or ad creative. And in that world, it makes total sense to focus on statistical significance. You’re just trying to figure out: is variant A better than variant B? If the difference is statistically significant, great – you pick the winner and move on.
But that doesn’t work when you apply it to incrementality measurement in marketing.
What you actually care about is:
- Is this channel profitable for us?
- Should we invest more here?
- What’s the incremental ROI?
- And how certain are we that additional investment will be a good use of the budget?
So instead of asking, “Was there a statistically significant lift?” you should be asking, “What’s the range of ROI values that are consistent with the data?”
Let’s say your test finds a 5% lift and it’s statistically significant. Cool. But what does that mean?
Well, you convert that into ROI terms, and maybe the midpoint estimate is a 4.5x ROI. But when you look at the uncertainty range, you see it goes from 1x to 7x.
If your business needs a 3x ROI to justify spend, this experiment didn’t give you the answer you need. Yes, there’s lift — but it might be unprofitable. It might be very profitable.
You just don’t know. And that is the real result of the experiment.
Here’s where most teams go wrong: they treat the mean as a guarantee.
They say, “It’s a 5% lift,” and bake that into the forecast like it’s locked in. Then they roll out the campaign, miss their number, and wonder what happened.
What happened is they ignored the uncertainty interval. That test wasn’t saying “you’ll get a 5% lift.” It was saying “you’ll probably get something between 0.001% and 9.5%.”
And because of how null hypothesis testing works, what’s actually most likely is that the lift was some small positive number, and your test didn’t have enough statistical power to give you a tighter range.
That’s the key takeaway: you should never just report the mean result and always look at and report the full uncertainty interval. If it’s a critical decision, you either assume the result will come in on the low end or you run a better-powered test to narrow the range.
2. Misinterpreting Lift Percentage Without ROI/CPA Conversion
We’ve had clients tell us:
“We ran an incrementality study and saw a 3% lift. It was statistically significant!”
But the reality is that the number is meaningless on its own because lift percentage doesn’t account for how much you spent to get that lift.
Again, to make a real business decision, you need to translate that result into something actionable — like incremental ROI or incremental CPA. Only then can you assess whether the channel is actually working for your business or not.
A 3% lift with huge spend might be wildly unprofitable. A smaller lift with efficient spend could be a winner. But you won’t know unless you do the math.
3. Ignoring the Range of Possible Outcomes
To go deeper on the above, we’ve also had customers come to us and say, “the experiment showed an incremental CPA of $75.” But then we dig in and see that the uncertainty interval ranges from $25 to $750.
That changes everything.
What the experiment is actually saying is: based on the data, the true incremental CPA could plausibly be anywhere in that range. The $75 is just the midpoint, it’s not the “most likely” number. It’s just the center of a very wide span.
And if you’re making budget decisions based on that midpoint, you’re taking a huge risk. You might be optimizing toward a result that never existed in the first place.
So when you’re looking at lift test results, don’t just ask: what’s the incremental ROI or CPA?
Ask: what’s the range? What’s the full distribution of outcomes this result is consistent with?
Because if that range includes both “this channel is killing it” and “this channel is lighting money on fire,” then the test hasn’t given you enough clarity to move forward with confidence.
4. Treating Experiments as One-and-Done
A common mistake we see: teams run a lift test, get the result, and think they’re done.
But that’s not how this works. Even if your experiment was perfectly run (and that’s a big if), the results start to go stale the moment the test ends.
Things change, and fast: the macro environment shifts. Your product changes. Your marketing mix evolves. Creative gets swapped. Targeting strategies adjust.
Let’s say your Meta prospecting campaign had a $45 CPA with a ±$5 range in April.
- After a few days, the results still mostly hold.
- After a month, you should already start treating them with caution.
- After three months? Depending on how seasonal your business is, those results might not matter at all anymore.
An experiment is a snapshot in time. It gives you a clean read — for that time. But the further you get from the test window, the less confident you should be that the results still apply.
The most effective teams aren’t treating experimentation as a box to check. They’re treating it as a continuous process:
You plan. You test. You validate. You update your model. You adjust your budget. You measure again.
Leading marketing organizations recognize that experimentation isn’t separate from measurement — it’s an integrated feedback loop that continually refines their understanding of what drives results.
TL;DR
- Statistical significance ≠ certainty
- Mean results ≠ guaranteed outcomes
- One experiment ≠ a measurement strategy
To make better decisions, focus on:
→ ROI and CPA, not just lift percentage
→ Full uncertainty intervals, not point estimates
→ Fresh, repeatable experiments — not one-offs
