1. Introduction
This article will sound counterintuitive – Recast is an MMM vendor (within our incrementality system). So why are we saying you should not outsource your model – entirely?
Because we’ve heard many stories of blind reliance on external teams which has resulted in misaligned incentives, unchecked errors, and results that failed to reflect the nuances of their business.
That said, of course, we honestly believe that working with a vendor is often the right option – if they meet specific standards of transparency and rigor. But more importantly, your organization must develop the internal knowledge to validate, interpret, and act on the insights the model provides. The key is collaboration, not dependency.
So let’s talk through this:
2. The Case for Working with MMM Vendors (and the case against working with them)
Why Vendors Are Valuable:
- Expertise: Vendors often have deep statistical and technical knowledge, which allows them to build sophisticated models. Building an accurate MMM is an incredibly complex endeavor. Over 30% of our team has a PhD in either math or statistics and we’ve dedicated many, many years of research and study to build our platform here at Recast. Without boasting, we wouldn’t be surprised if we had the most complex Bayesian model in the world.
Sure, you can run numbers through Excel and call it an MMM or use an off-the-shelf tool – but they’ll most likely be wrong. When you’re using an MMM to help allocate millions of dollars in the budget, you can’t afford to not use the team with the most expertise. - Efficiency: A good vendor should make the process of using MMM as straightforward as possible. They handle the technical heavy lifting, and your team gets to focus on strategy and execution.
However, there are outsourcing risks, especially if you take a “set it and forget it” approach.
When Outsourcing Goes Wrong:
- Vendors Don’t Know Your Business Like You Do: External vendors are never going to know your business the way that you do. You have more context on the overarching business strategies than external parties do.
- Misaligned Incentives: You run the risk of putting into place bad incentives where they’re incentivized to tell you what you want to hear. If a vendor knows that no one internal to the organization actually understands what’s going on, the vendors can end up incentivized to fudge the results.
3. The Role of Transparency and Validation in Vendor Relationships
With any external MMM vendor, please make sure you deeply understand the technology and what assumptions are being made.
We’ve said this often but it’s important: you need to have a way to check and validate what the model is giving you. Wiping your hands off completely can really backfire.
Without visibility into the assumptions and methodologies behind the model, you’re flying blind – and that’s risky when millions of dollars are at stake.
What Good Vendors Should Provide:
- Clear Documentation: Vendors must explain how the model works, the priors used, and any assumptions baked into the results. We have made ours available to the public.
- Validation Tools: Vendors should offer ways to test the model’s accuracy and consistency, such as out-of-sample forecast accuracy checks and lift test alignment.
- Honesty About Limitations: No model is perfect. Vendors should be upfront about the uncertainties in their approach and what the model cannot do.
Questions to Ask Your Vendor:
If you were to write down the 10 most important features of an MMM, the first 9 would all have to do with evaluating the accuracy of the model and how to validate it.
The main thing that you should care about from the model is that it gets you the right result. Everything else that an MMM might do does NOT matter if it’s not actually measuring marketing effectiveness correctly.
The problem is that every vendor will say “Our model is the best and of course, it’s perfectly accurate”. So asking them doesn’t really help you. But you can start by asking them these:
- What assumptions are driving this model? For example, what priors are set, and how do they align with your business context?
- How does the model validate its results? Look for evidence of rigorous testing, such as comparing results to lift tests or using stability loops.
- What happens if the model is wrong? A good vendor will be proactive about identifying risks and providing solutions.
4. Why Marketers Must Build Internal Knowledge
If your marketing organization lacks technical staff, you should address that quickly and upskill your current employees.
With fairly technical people, the results from a media mix model should be understandable. They will still need training to interpret the specific metrics and results, but they should be able to use an MMM effectively.
Another option is to pull people from other teams, like finance or data science. Both teams tend to have an intuitive understanding of media mix modeling and can provide support to your marketing team.
The best approach to MMM is a collaborative one, where vendors and internal teams work together to maximize impact.
The internal team knows their organization very well and they’re able to use Recast, take the results, and communicate them internally to actually drive business change for our customers. They understand how decisions are made and how to move things forward. That’s what we’ve found to drive better decisions, higher confidence, and, ultimately, better results.
As a final note, vendors also have the responsibility to help marketers have an optimal UX.
For example, we take a lot of pride in being research-focused and compute-heavy. But we also understand that we’re communicating with marketers and they have to use the model to make day-to-day decisions about where they’re going to allocate spend. And they also need to communicate their rationale to their own stakeholders and executive teams.
So, while we’re scientists, they’re often not. This means that the results of our model need to be broken down into digestible insights and recommendations that are easy to understand, use, and relay within their organization.
That’s why we say Recast is built *by* scientists *for* marketers.
