As more marketing teams understand the need to add new measurement methods beyond digital tracking, we expect more and more emphasis on MMM and experiments.
Experiments and MMM can be very complimentary – but they have to be done well.
One of the challenges with both methods is that they’re time-consuming and take time to set up – especially when compared to the immediacy of Google Analytics or social’s in-platform reporting.
Transparently, that used to be fair criticism. Traditional MMM, for example, used to mean getting a report every six months by whichever consultancy you hired, and, by then, the data was outdated.
Modern MMM has disrupted that notion – Recast’s MMM updates weekly so you can avoid surprises and have more actionability and capability to optimize in-flight. It’s also easier to validate and more transparent and trustworthy.
But what about experiments? How can they be done quicker and more often?
We just had a Recast Measurement Coffee Break with our co-founder Michael and Haus’ CEO Zach Epstein and the topic of high-velocity testing came up. Here’s a summary of what they discussed:
What is high-velocity testing and where should you use it?
Marketing bets operate on a spectrum, and multiple factors will define how you set up each experiment: the insight you’re trying to drive, how much money is on the line, the importance of the network, etc.
Zach explained that, only a couple of years ago, he didn’t believe companies other than Netflix or Booking.com would ever run several experiments per month. Now, however, he sees plenty of companies doing that – which shows how companies are going beyond MTA.
The goal behind high-velocity testing is to help brands make ROI risk-adjusted highly probable bets.
Instead of taking one/two quarters to run one experiment – which tends to be the standard, modern teams will be asking themselves what is the right two-week / four-week experiment we can run.
Where can we get away with 80% statistical power? Where can we get away with 70%? What is important enough that we need +90%?
Not everything needs to be 95% power – but you still want to do it in an ROI-plausible way where you’re to have concrete enough outcomes.
A practical example: high-velocity testing and seasonality.
Let’s consider seasonality: here’s how we approach it for MMM at Recast, but what about when it comes to experiments?
To test during seasonality, you need to be able to experiment and get insights quickly. For example, if you’re a snowboarding equipment company, your high season will be October and November. You can’t wait six weeks to run a GeoX test – which is what they tend to recommend since they’re optimizing for outcomes and want to demonstrate a lift no matter what.
Instead, if you do high-velocity testing you’ll get signal within a shorter window of time, and you’ll be able to actually play out the learnings you get from the experiments. You might sacrifice a small degree of certainty, but you’ll win time to execute,
How modern markers manage uncertainty:
We’ve all heard that 50% of marketing is wasted and you don’t know which half. Zach argues that this might be even worse since platform reporting creates conversion-optimized signals towards non-incremental activities.
At Recast, we understand uncertainty really well because there’s no getting away from it. A lot of marketers, especially those not trained in statistics, can struggle with this because they’ve been trained to expect certainty.
You log into Meta and you see the CPA number with no uncertainty associated with it at all. It’s an uphill (but necessary) battle to show marketers why they need to distrust those numbers and how to make good decisions in the face of uncertainty.
The platforms that express uncertainty are the ones that are being more truthful than the ones that are optimizing for some point estimate – which isn’t actually reflecting that true underlying uncertainty. They just don’t expose it to the user.
Over the last ten years, we’ve conditioned marketers to just want a number. But there have been fundamental shifts in the marketing measurement world that are forcing us to rewire that thinking and adjust to MMM and high-velocity testing.
Triangulate:
We’re not saying testing is the holy grail of marketing measurement. Experiments should be part of your portfolio. MMM should be part of your portfolio. Even digital tracking should be part of your portfolio.
They all have a time and a place in the sequence – all these pieces can be complementary and come together through triangulation. That’s when you’ll start to get a clearer picture of what’s incremental and what’s not.
