Imagine this: your MMM refreshes, and suddenly last week’s top-performing channel is now your worst. You had just told your CFO that Meta was your most incremental channel and we should increase our budget there because your MMM told you that… and now the model is saying the complete opposite.
You go and update your CFO, and, of course, they want answers. Your media team panics. You wonder whether to go back on all your budget decisions. This is a huge modeling red flag.
When model results change dramatically with small changes to assumptions or the underlying data, it means the model doesn’t work, and its recommendations shouldn’t be trusted.
Yet this is more common than most marketers realize.
Some vendors “freeze coefficients” to maintain consistency. That also does not work, and it’s a really bad scientific practice. They’re effectively locking in the model’s assumptions and ignoring new data. It hides instability, but it doesn’t solve for it. Over time, the model is further and further from reality. And this issue gets even more dangerous as models refresh more frequently – weekly vs yearly.
For any MMM to be actionable, it has to provide insights that remain consistent over time unless the world genuinely changes. So – how should you measure and detect model instability?
What Stability Loops Are (and Why They Matter)
That’s where Stability Loops come in.
Stability Loops are a validation method that Recast uses to make sure that a model’s insights hold up over time. They simulate the exact conditions of weekly model refreshes, adding one week of new data at a time and quantifying how much the model’s outputs change.
Here’s how they work: for every parameter in the model – say, the ROI of non-brand search – we calculate the uncertainty interval from the prior run and the current run. Then, we measure the overlap between those two intervals. Stable estimates week to week are evidence that our model is picking up on a consistent signal from the data rather than overfitting to noise.
But not all parameters are equally important. The bigger the impact a parameter can have on your model’s predictions, the more concerned we are with it being stable. That’s why Recast weights the stability calculation based on how much each parameter contributes to the model’s dependent variable.
Your model should learn from new data. But one week of data out of a hundred shouldn’t cause wild swings. If it does, the model is unstable.
Why Most Teams Don’t Do This – and Why You Must
If model instability is such a critical risk, why isn’t everyone testing for it?
Simple: many teams don’t want to look. Some avoid it entirely by only updating their MMM once or twice a year. And, at that frequency, instability is easy to ignore or hide.
Others go further and “cheat” stability. That includes freezing coefficients between model runs, applying overly restrictive priors, or overriding outputs manually to make them “look right.” That’s performative theatre.
And yet, these practices remain common because real stability testing adds accountability. If your MMM can’t handle weekly refreshes without falling apart, you have to fix it or admit it’s not ready for production. Many vendors would rather not open that door.
At Recast, we believe the stakes are too high to ignore this. An unstable MMM creates false confidence and leads teams to shift millions in budget based on noise. It also undermines every future insight it generates and breaks the trust chain with the people who actually use the model.
That’s why we build Stability Loops into every production model by default. No shortcuts. No coefficient freezing. Just a clear, repeatable way to know if the model is picking up real signal or not.
If you’re evaluating MMM vendors, go and ask them: “what happens to your estimates when you add one week of new data?” If they don’t have a real answer – or worse, if they mention freezing anything – keep looking.
TL;DR:
- If your MMM gives dramatically different results each time it refreshes, it isn’t reliable and shouldn’t guide budget decisions.
- Stability Loops test whether your model is learning from real signal or just reacting to noise by simulating week-by-week data updates.
- Recast measures stability by calculating the overlap in uncertainty intervals across runs and weighting results by business impact.
- Common shortcuts like freezing coefficients or infrequent updates may hide instability – but they don’t fix it.
