Media mix modeling (MMM) isn’t theoretically hard to do, but it’s really hard to do well. Every step of building an MMM is a potential place for it to break, inject bias, and can lead to inaccurate results. One of these “steps” we find that is often not discussed enough and carefully executed is right at the very beginning, during the configuration process.
At its core, model configuration is about making sure your MMM reflects the true dynamics of your business. And let’s be clear – poor configuration means an untrustworthy model, biased results, and inconsistent insights. You’re not giving MMM a shot at being right if the foundation isn’t done well.
For example, imagine an MMM that predicts your paid search ROI is negative – which we all know defies logic and common marketing principles. Without proper configuration to guide the model, these kinds of anomalies are all too common.
This article will cover how we at Recast approach it with rigor as our core value.
Recast’s Rigorous Configuration Process
We’ve had people ask us why Recast’s model configuration and validation process takes multiple weeks. It’s because we’re obsessed with validating the model rigorously – and that takes time. But it’s also what matters most.
We obviously could just run the data through a model and have results in a few hours. But just getting results isn’t enough: the real work comes in validating the results and making sure the model actually passes validity and robustness checks.
So, this is what Recast’s configuration process looks like to make sure our model is rigorous:
Step 1: Setting Parameters
The first step is defining key parameters specific to your business. Every company is different, and our assumptions must reflect your unique dynamics.
- Organic Demand: For established brands, organic sales might account for 70-80% of total revenue. For newer brands, this figure might be closer to 20-30%. Accurately estimating organic demand sets the baseline for your marketing’s incremental impact.
- Channel ROIs: By analyzing your total revenue, organic demand, and marketing spend, we calculate an expected ROI range for each channel. This makes sure results are internally consistent.
- Time-Shift Curves: Different channels drive results on different timelines. TV campaigns might impact sales over weeks, while paid search delivers immediate results. We estimate these time shifts to help the model capture each channel’s unique dynamics.
Remember: no statistical models are “assumption-free”. Either you hide the assumptions or you make them explicit, but there’s no escaping the assumptions.
In the case of a media mix model, which is naturally very complex, you’ll need to make lots of assumptions and “configure” the model in a way that matches your business and the business questions you’re most interested in answering.
One thing to note: it’s important to do this configuration before actually seeing the model’s results! We want to start with our assumptions, and only then run the data through the model.
Once we see the results from the model, we might be incentivized to go back and change our assumptions to make the results match what we want them to be. This is bad science and can lead to biased models that will lead you (and your company) astray!
Step 2: Prior Predictive Checks
Once parameters are set, we run a prior predictive check to validate our assumptions. This involves simulating the model using only the priors –without allowing it to learn from the data.
Example: if a prior predicts your sales could range from $1M to $10B, we know something is off. This test helps catch such issues early and shows us if the model is starting with realistic boundaries.
We’re not asking the model to learn anything yet – we just want to see if the model can simply return the assumptions we provided.
Step 3: Parameter Recovery Testing
Most of the metrics we care about—like ROI or time shifts—can’t be directly observed. To make sure the model estimates these accurately, we use parameter recovery testing:
- Simulate a dataset with known “true” parameters.
- Feed the simulated data into the model and see if it recovers the correct parameters.
We’re looking for the model to find the right answers, not just any answer that fits.
This is our most crucial test!
This process gets repeated hundreds of times under different scenarios to expose the model’s blind spots. If your MMM can’t reliably recover key parameters (like channel ROI or time-shift effects) on fake data where you know the truth, it won’t fare well on real-world data where the truth is unknown.
Step 4: Holdout Testing
We run tests where we hold back data (30, 60, 90 days) and see how well the model predicts those missing periods.
A good scoring rule should look at more than just how close a forecast is to the actual value. It should also consider how confident the prediction was, which we can usually tell by how narrow the forecast range is.
If the model’s predictions are consistently off or overly broad, we know it isn’t ready—and we iterate until it can reliably predict holdout periods with high accuracy.
Step 5: Stability Loops
Finally, we test for consistency with stability loops. By simulating weekly data refreshes, we check whether the model’s insights remain stable over time.
For instance, we make sure it doesn’t tell us Facebook is our best channel one week and our worst the next without a legitimate reason.
Final Thoughts
Anyone can build a media mix model in Excel or Google Sheets. With a decent tutorial, you could create one in under an hour. The problem? It would almost certainly be wrong.
We’re only confident a model is ready for the client to use if it has passed all our checks for accuracy, consistency, and actionability.
It’s this process of iterating through all of the validation checks to know the model actually works and can be used for real decision-making – that’s the part where you don’t want to skimp on time and effort.
Because you don’t just want an MMM. You want an MMM that is actually accurate. And you want to be able to prove it.
