We see a lot of media mix modeling RFPs here at Recast. Unfortunately, a lot of them don’t emphasize what we, as practitioners, think is most important – not to us, but to them as a brand.
There are 3 elements that should be on every MMM RFP:
- Verifiability of the model
- Model operations
- Usability of the results
Each element should be a section in the RFP and include some follow-up questions, so let’s expand each one:
1 – Verifiability of the model
The first thing that you should be thinking about is verifiability and how are you going to validate the results are actually correct. This is the thing that we talk a lot about at Recast, but it’s that important and what we think you should put the most emphasis on in your RFP.
You want to ask every vendor you meet how transparent are the results going to be and how –specifically– are you going to be able to verify and validate them.
Methodology questions are what we see the most, and they are useful and interesting. But really, what’s actually the most important takeaway is going to be the validation and the verification procedure that you’re going to follow in order to check the results are right once they’re delivered.
2 – Model operations
You also want to think about all the operational components of your model. Make sure you’re asking questions like:
- What’s the refresh cadence going to be?
- How often are you actually going to be re-estimating the model and re-estimating all of the models’ parameters?
- What’s the procedure for identifying when things have changed?
Beware here – a lot of MMM vendors do this confusing thing where they have different types of refreshes, or they call it a rebuild which then means something different than a refresh. We don’t really understand these different terms, but a lot of these vendors use them.
We’ve even heard stories from people who were told their MMM refreshes every week but they’re just multiplying the parameters by the new marketing activity that’s coming in every week, which you could do in Excel – and the model actually refreshes once a quarter.
Put simply, you really want to be clear on exactly what’s happening.
3 – Usability of the results
A model that is not made to be used is a bad model – plain and simple. Here, you want to ask questions like:
- How are the results actually going to be used?
- What is the platform that you’re going to use to understand the results?
- What underlying data are you going to have access to?
This can get really painful with some vendors where, every time you want something, you have to ask a question to the support team (and they’re very slow), and they have to ask their team in some foreign country to pull the data for you.
If you want to use MMM in a fast-paced, ongoing operational way – on-demand, as we say at Recast – great usability is what unlocks it. What you really want to see are platforms where you can self-serve and you don’t have to wait on someone else.
Whether you use Recast or not, we’d recommend including all these questions in your MMM RFP. They should give you a better understanding of 1/ how to validate the model, 2/ how the model operates on what matters the most, and 3/ how easy to use their model is.
If you’d like to chat and see how Recast answers these questions, we’d love to chat.
For extended reading and even more questions to ask your MMM vendor, this can help.
