With digital tracking breaking, consumer brands that relied on multi-touch attribution are now looking for alternatives to measure marketing effectiveness. The two methods that comply with privacy regulations and can help get closer to true incrementality are marketing mix modeling (MMM) and conversion lift studies (CLS).
Quick context on both:
Marketing mix modeling is a statistical modeling technique that marketers use to determine which channels in their marketing mix deserve credit for sales, in order to reallocate budget to the highest-performing areas.
Conversion lift studies are tests run by marketers to validate what performance would look like if you switched a channel off, or scaled spend up or down, usually comparing outcomes against a control group. Geo-tests, holdout tests, and platform-run randomized control trials (like Meta’s Conversion Lift) all fall under this umbrella.
MMM and CLS don’t operate in silos, and we highly recommend using them together to get a more clear picture of channel performance. They are very complimentary when done correctly. So, let’s talk about the three ways lift tests fit in within MMM:
#1 – Use lift tests to calibrate your MMM
Lift tests are a great way to calibrate your MMM – they give the MMM more information about the ground truth, which will help the model settle on the set of plausible parameters that are consistent with the results that you got from the test.
This can be tricky because the vast majority of MMMs make the assumption that marketing performance doesn’t change over time. If you have two different lift tests with two inconsistent results from one channel, which is very common because channel performance changes over time, it’s not clear which of those lift tests you should use when you’re calibrating your MMM results.
For full transparency, the industry doesn’t have great solutions for that in static MMMs. Those platforms are making the assumption that channel performance is constant, so two lift tests with different answers force the modeler to pick one and discard the other.
Since we have a Bayesian time series model, what we do at Recast is estimate what the incrementality of every marketing channel is – for each day. That lines up really well with the way lift tests work because we can incorporate those results directly into the Bayesian statistical model by putting priors on the performance of that channel, but only at the time when the lift test was run.
We’re treating the evidence correctly by considering it a snapshot in time and not applying it to how that channel has performed over all of history.
#2 – Use lift tests to proactively test your MMM
Another good way to leverage lift tests is to use them to validate the MMM out of sample. You should be able to show outside your MMM that the MMM is right (or wrong) — that’s something we’ve built into how we work with clients to align our incentives with theirs. After you receive the results of your MMM, run a fresh lift test (one that wasn’t already fed in as a prior) and check whether the result lines up with what the model predicts for that channel and time period. If it does, you can be more confident the MMM is picking up real causal signal.
If we get results from the MMM that say Meta is our best channel and radio is our worst channel, we should be able to test that the following month. A lift test will help you see if the model continues to predict accurately and if your overall marketing efficiency actually improves. Recast’s MMM updates weekly so it gives you the opportunity to be more dynamic and run lift tests more often to audit the model.
A quick note here: when an MMM and a well-run experiment disagree, the well-run part being very important, it’s normally the MMM that’s wrong. The model is making a lot more assumptions than the experiment so when you have experimental data that conflicts with your MMM, go take a look at why the model might be incorrect.
#3 – Use MMM to get additional insight into incrementality
Another way to think about the relationship between MMM and CLS is to think about MMM as providing additional insight into incrementality for all of the times that you can’t be running lift tests.
Lift tests are a great way to get to incrementality, but they have a couple of drawbacks:
Firstly, they are expensive to run so most brands do them sporadically. In our experience, companies typically have some discrete points in the year when they test a few of their different channels – but it’s only one or two tests a year. Secondly, because they are measured at discrete points in time, they tell us about the performance of the channel at that time, but not necessarily about the performance of the channel in general.
Let’s say you ran a lift test six months ago. You got information from it, but we expect the marketing world to have potentially changed in meaningful ways over the last six months. MMM can pull all of those different pieces of evidence that you have together into one view based on all of the current data that you have.
Your marketing mix modeling framework should be able to take into account the evidence from lift tests, but only as a snapshot of that point in time. We want the lift tests to help the model gain signal, not bulldoze its estimates.
How to combine lift tests with Recast’s model
Our models don’t propose lift tests directly but our clients may see channels where MMM and digital platform tracking disagree. Instead of arguing over methodology, we suggest running a lift test and verify which method is right. They often see that the lift test looks like the Recast results and they now feel confident that Recast is measuring incrementality.
Another situation where this can happen is if Recast’s results are very uncertain – maybe our client has a few small channels or channels that are highly correlated with other channels. We show the uncertainty and, if we want to get more precision, we propose running a lift test.
At the end of the day, these marketing measurement methods are very complementary, and when done right, you can use the MMM to pull together the evidence that you have from those lift tests in order to get a holistic, continuous view of incrementality for your business.
