Recast’s MMM uses Bayesian statistics, but why doesn’t everyone else?
We asked this to Recast’s co-founder Michael Kaminsky, and the rest of this article covers what he answered:
“When I started my career as an econometrician, I was pretty comfortable with the frequentist methods that people who did a technical degree saw in their statistics courses: null hypothesis testing, P-values, etc.
But when I first started learning about Bayesian statistics… I was incredibly skeptical. It didn’t seem rigorous nor unbiased – “this prior thing, you can put in whatever numbers you want and you’ll get back whatever you put in.”
So, I truly empathize with people who start to learn about Bayesian statistics and are skeptical.
That being said, once I actually really put in the work of understanding the theory behind it and the power of the tools that are available to Bayesian statisticians, I changed my tune.
I now very much appreciate what they can open up from a statistical analysis perspective. Bayesian Marketing Mix Modeling is at the forefront of how marketing attribution is done by modern consumer brands like Away, HelloFresh, MasterClass, Truebill, and Harry’s.
But I also understand why some can be unsure if this is the right method for their MMM. Other than requiring a thorough study to see its statistical validity, I often see people question this:
Do you get the same results with a Bayesian approach as with a frequentist/traditional approach?
In the most simple cases where you have experimental data or you have an A/B test, you’re not going to get different results between any of these methodologies. You have a clean experiment and you’re just trying to compare the average treatment effect between two groups – Bayesian statistics won’t add much there.
Where Bayesian statistical models can be really powerful is in the process of specifying upfront a data-generating process and then a set of priors about what’s realistic and what’s not. This methodology allows you to express very complex models and get reasonable inferences back from them.
These can be based on numerous considerations, such as:
- Positive correlation between Facebook Ad spend and sales.
- Anticipated ROI for Facebook lying between 1x and 3x.
- The principle of diminishing returns in ad expenditure.
- Latency in the impact of ads on sales.
By harnessing priors, implausible or extreme models can be ruled out, allowing for a narrower, more realistic focus.
When I was trying to do similar work with frequentist economy metrics, we often had to cobble together a bunch of different methods that weren’t purpose-fit for the question that we were trying to answer.
We got okay results, but it felt like a hack.
If you just use linear regression, you will often get biased estimates – especially with variables dealing with multicollinearity. We’ve heard funny/sad stories, like spending money on Meta retargeting ads actually detract from sales. It’s basically impossible you’ll get a negative ROI from that, but that’s what running the OLS model on correlated variables might show you.
With Bayesian statistics and specifying a data-generating process that you can estimate with a tool like Stan or PyMC3, you can express exactly the model of how the world actually works. And then you can use the power of Hamiltonian, Monte Carlo, or MCMC to get inferences back that match exactly the question that you’re trying to answer.
Instead of asking “What are all the combinations of ROIs that would have led to these sales figures with past levels of spend?”, you can ask the slightly more complicated question “What are all the combinations of ROIs that would have led to these sales figures with past levels of spend, assuming that ROIs are never below zero?” Or even, “assuming that ROIs are never below zero, and are very rarely above 10x”, and so on.
The fusion of prior knowledge and empirical data makes Bayesian approaches particularly enticing for modern marketers. When you’re striving for results that mirror real-world events, Bayesian Marketing Mix Models shine. They allow for the integration of hands-on expertise, resulting in more plausible outcomes. The real-time updating capacity, sans human intervention, makes it very valuable.
That’s one of the key reasons why Recast uses Bayesian – it’s a really powerful tool, especially in circumstances where you have observational data in a complex causal model or data-generating process.
That said, while Bayesian empowers Recast’s model, it doesn’t solve everything. Even though our heart is in the right place, we recognize our own biases. We’ve built in the features that help our customers to hold us accountable outside of Recast: out-of-sample forecast accuracy and validation through lift tests. That starts to align incentives and our customers are more confident taking those insights and actioning them.
