The math and science part of media mix modeling is extremely hard to get right. But even when you do, many MMMs will still fail because the system around them rewards the wrong outcomes.
There is enormous pressure in marketing measurement to show that things worked. Marketers want to prove their spend is driving value. Internal analysts want results that land well with stakeholders. Vendors want renewals and long-term relationships.
None of this requires bad intent. But it does bias outcomes.
This article will cover the common flawed incentives that will corrupt your MMM and how to set guardrails so that it doesn’t happen.
Why Most MMMs Are Optimized to Tell You What You Want to Hear
Let’s start with something we all know: in a healthy measurement system, you should expect to see losers. You want to find the channels, tests, and ideas that didn’t work, because that’s where learning actually happens. A system that can’t surface uncomfortable findings can’t be trusted to surface real winners either.
This problem isn’t unique to marketing. Academia went through a replication crisis for the same reason. Researchers were incentivized to publish “significant” results, so the system quietly optimized for positive findings instead of the truth. Years later, many famous studies turned out to be wrong because the incentives rewarded publishable results over accurate ones.
Marketing measurement has even worse incentive alignment:
- A common failure mode starts with optimizing for historical fit. A team builds an MMM, looks at in-sample accuracy, and sees a great result. The model explains the last quarter beautifully. Channels line up with intuition. Everyone feels good.
But historical fit is easy. If you give a model enough flexibility, it will always find a story that explains the past. The harder question is whether it can predict the future accurately – and that’s the only model worth trusting (and paying for!).
- Another example: vendors (or internal teams) running dozens or hundreds of model variations, then selecting the one that “looks right.” Maybe it shows a strong lift for the brand campaign that leadership already believes in. The analyst didn’t (technically) fabricate anything – they just chose the result that felt most reasonable to them. That’s still grading your own homework.
- This also happens with experiments. A lift test shows a statistically significant result, so everyone calls it a win and they increase the budget for that channel… but the expècted revenue never materializes.
We like to think that most of this happens unintentionally. People genuinely believe the results. But it doesn’t matter – without guardrails that take away biased incentives, your model will break.
Falsifiability: The Line Between Measurement and Storytelling
At some point, every measurement system has to answer a simple question: can it be proven wrong? That’s what falsifiability means in practice – can the model make a prediction that’s specific enough to test in the real world, and then be held accountable when reality shows up?
For example, if your MMM says non-branded search has a 10x ROI, that’s testable. Go raise bids and budgets. See if you drive incremental revenue anywhere close to a 10x ROI. If you don’t, the model is wrong. Full stop.
It goes back to the feedback loop that marketing teams should be following:
Forecast → Act → Observe → Validate
This matters because these checks can’t really be gamed. Either the model’s forecasts come true… or they don’t. If your measurement can’t be wrong, it also can’t be trusted. The moment you introduce falsifiability, you start optimizing for what’s actually true vs what feels good.
The Guardrails Recast Uses to Keep Incentives Honest
At Recast, we know that incentives will win unless we build explicit guardrails to keep ourselves honest.
The most important one is out-of-sample validation. Every week, we take models that were deployed in the past and ask: how well did they predict what actually happened? We test performance on data the model has never seen before. Not “how well can you explain last quarter,” but “did your forecast hold up?”
The model makes a claim about the future. Time passes. Reality happens. And the model is judged on whether it was right. And this happens over and over again.
We’re also extremely careful about information leakage. We exclude variables the model wouldn’t have known in advance – things like branded search or affiliate spend – because including them would artificially inflate accuracy. It might look better in a demo, but it wouldn’t be honest.
And we don’t keep these checks private. We publish accuracy metrics and share them with customers. This makes it impossible for us to put our thumb-on-the-scale and forces us to be accountable.
As a final note, you should not fully outsource the interpretation of your MMM — not even to Recast.
First, no external vendor will ever have full context on your business. They don’t sit in your revenue meetings. They don’t own your pricing strategy, product roadmap, or risk tolerance.
Second, again, blind trust creates bad incentives. If no one internally understands how the model works or how to pressure-test its outputs, vendors can end up incentivized to fudge the results (even if unintentionally).
You don’t need to be a modeling expert, but we do recommend building enough internal literacy to ask the right questions and challenge results when they don’t line up with reality.
