Marketing mix modeling is hard. Trust us. When we were starting Recast, we thought it’d be easy: we’d use some off-the-shelf Bayesian time-series models and six months later we’d have a top-shelf model. But unfortunately, it didn’t work like that. It took us multiple years of PhD-level research time to get to a model that has tolerably good results, and we’re still uncovering new edge cases to handle every week.

In this post I’ll talk a bit about why the problem is so hard and what makes it different from other more tractable problems that data scientists are more accustomed to working with.

Not just prediction

It’s not enough for an MMM to be able to predict sales. In fact, just predicting sales in the future would be a relatively easy task. What makes MMM so hard and so easy to get wrong is that the model needs to accurately model why sales would be at a certain level. If you get the why wrong, then when you make changes to your spending patterns, your model will suddenly stop working. 

This is another version of the “correlation does not imply causation” problem. Several channels may be correlated with sales, but they may not cause sales (at least to the same degree). A model that only has to predict sales only has to model correlations; an MMM that will be right even when you take action based on its recommendations has to model causation, or the true incrementality of each marketing channel. 

This means that a lot of the developments in machine learning over the last 10 years aren’t applicable to this problem. Most ML tools are designed for “pure prediction” problems where capturing the internal correlations in the data are all that matters, but they don’t put any emphasis on the “why”. So if you run your MMM model through a neural net or XG-boost, you might get very good predictions, but you won’t get an accurate model of how sales will change once you actually adjust your marketing budget.

How we solved it: At Recast, we put all of our focus on getting the statistical inferences correct using our proprietary fully Bayesian model. We use out-of-sample prediction to guide us, but our focus is on building a model that reflects reality and produces reasonable inferences that can be validated experimentally.

Small data

For an MMM, Recast is comparatively greedy, requiring daily-level data in both the marketing channels and the outcome variables (usually sales or customer acquisition). Even still, with about two years of data (typical for our customers), that is only about 760 rows of data. 

This quantity of data is simply way too small to apply highly complex ML algorithms (which, even if you could apply them, wouldn’t be fit-to-task because of the causation issues above). 

It gets worse: because the data is a time series, each new day of data is typically pretty similar to the day before. So the 760 or so rows of data in your dataset do not amount to 760 unique pieces of information in the same way that, say, a dataset of 760 customers would. 

But the amount of data is only an issue relative to the number of parameters you want to estimate — or the number of things that you want the model to tell you. And unfortunately, that list is very long, as the next section will detail. 

How we solved it: At Recast, we use a fully generative Bayesian statistical model estimated with the latest techniques in Hamiltonian Monte Carlo (via Stan). This allows us to flexibly estimate a huge number of parameters with small data that wouldn’t be possible otherwise.

Complex, unobservable process

MMMs are full of interacting nonlinearities and complex processes. To take one example: when you spend money today, some of that effect will happen today, but some of that effect will linger until tomorrow, or the day after, and so on. We call this the time shift, but in the literature this is sometimes called the adstock. 

Another one to account for is diminishing returns. Let’s say your MMM identifies a promising channel. How much money should you spend in that channel? Without accounting for diminishing returns, the model will tell you to spend as much money as you have! These models will tell you that your efficiency in a channel is the same at $1k, $100k, or $1mm of spend. This is counter to every marketer’s experience, and soon leads to absurdities. This means that the effect an MMM assigns to a channel needs to decrease with the amount spent in the channel. We call this “saturation” in Recast. 

Finally, what about sales? Or holidays? How does a marketer think about a particularly big day of sales on December 24th? It’s often the case that sales and holidays are big days, but partially at the expense of the days before and after, as customers “push back” or “pull forward” their purchases. Modeling this correctly entails superimposing a complicated nonlinearity on the rest of the sales curve. 

How we solved it: At Recast, we’ve spent years on R&D building a statistical model that’s flexible enough to handle all of these situations. We’ve integrated feedback and suggestions from clients over the years to build a single model that captures all (or almost all) of the complexities that crop up in marketing contexts.

Different types of questions to be answered

Internally developed MMMs often suffer from a too-many-stakeholders problem. The core issue is that MMMs are often asked to do too much: estimate the but-for scenario where there was no global pandemic, or perfectly predict sales next quarter assuming that we increase prices or generate an optimal budget for the marketing program. All of these are totally valid questions to want to ask, but they’re very different and no one model is going to be able to answer all of them correctly.

In practice, what happens is modelers get pulled in so many different directions that they end up building a franken-model that tries to address all of the different questions but doesn’t answer any of them correctly.

How we solved it: At Recast, we’re narrowly focused on answering the incrementality question: if you have an additional $100 in marketing budget to spend, where should you put it? This allows us to focus on what we believe is the most important question to marketers and answer that question really well.

What Else Are We Thinking About?

There are a number of unsolved challenges in the space that we are currently investing R&D resources into at Recast. These include:

• How to correctly model the interaction effects between channels like “retargeting” and “branded search” and the rest of the marketing program?
• How to correctly model the incrementality of contractual affiliate marketing programs?
• How to handle variance in creative performance for live-read channels like Podcast and influencers?
• How to handle geographic-level differences in marketing, price, and promotion?