Is AI revolutionizing media mix modeling?
People have been asking me about how AI is changing media mix modeling, and the answer really depends on exactly how you define “AI”.
The Role of Large Language Models in MMM
Large language models, including ChatGPT, don’t really have a lot of applications to MMM.
Media mix modeling is a causal inference problem, and LLMs cannot currently do this type of analysis or statistical modeling correctly or effectively.
The only applications I see (right now) in the context of MMM are for things like helping you summarize results or reading documentation and explaining back its assumptions.
While I’m generally bullish on LLMs and their impact on productivity, we have seen them get subtle but important technical things very wrong. Be very careful about how you use LLMs for statistical analysis and what the implications of their error rate are.
Now, if you define AI more broadly where it includes other types of machine learning, then yes – that very much applies to the world of MMMs.
Under the hood, Recast uses a fully Bayesian statistical model that is estimated using a method called Hamiltonian Monte Carlo (HMC). There is a very real sense in which this algorithm is doing “machine learning” or even “artificial intelligence”. (You’ll notice of course that we’re allergic to hype at Recast so don’t use “AI” in our marketing materials.)
Whether you should call that artificial intelligence or not, I don’t really know. It is a type of machine learning, and it allows us to estimate really complex models very efficiently – which was just not possible 15 or 20 years ago.
When we look back at your marketing activity, there will be some days where there are different amounts of marketing activity happening and there are different amounts of new customers that you’re acquiring, or revenue that you’re generating, or new marketing qualified leads you’re bringing in.
We use AI and machine learning to find the relationships in the historical data. On days when you spend relatively a little bit more on Facebook, how many additional new customers do we generate? On weeks when the podcast drops, do we tend to see an increase in marketing qualified leads?
Of course, there are a lot of other sophisticated things that are going on in our models. But the idea is that we’re going to look at the days where your channel mix spend changes and how they’re different, and then we’re going to look at how we acquire different amounts of customers based on different amounts of marketing activity.
We are taking what consultants used to do by hand: running hundreds or thousands of different model specifications and then evaluating which model made the most sense or had the best metrics but, instead, we’re using AI and machine learning to do that in the cloud without human intervention.
But the key thing to remember is that media mix modeling is a causal inference problem and the current generation of LLMs don’t have the ability to do that effectively.
And that’s okay. I love LLMs for what they’re good at. They’re just not good at this particular thing. So be careful when you see AI for MMM, and think hard about what exactly is the AI doing and how you are going to catch and prevent fundamental errors.
Considerations When Choosing an MMM Vendor
It’s tempting to leverage the hype surrounding AI for branding purposes, but this often leads to misconceptions and misguided expectations. Many businesses, eager to appear cutting-edge, might market themselves as “AI-driven” without fully understanding or utilizing the technology’s capabilities. Misusing AI in this context can result in not only flawed budget allocation strategies and wasted resources, but ultimately, a loss of credibility in MMM itself.
If you’re considering an MMM vendor, we recommend not falling in love with AI branding and really get to the core of how AI is actually being used in the model.
And, as always, make sure you’re asking them the right questions to vet them – whether they use AI or not. Questions like:
→ How can we validate the model’s results?
→ Which parameters in the model are estimated versus those selected by an analyst?
→ For parameters set by an analyst, what is the process for setting those parameters?
→ Can I easily obtain confidence/credible intervals for all parameters estimated in the model?
→ Are we able to see true holdout predictions based on the model? Can we monitor those over time?
→ Is there any pre- or post-processing happening in the pipeline that’s influencing our model’s results? What happens if we don’t trust the results?
