The future of marketing measurement goes through understanding how to use multiple methods in conjunction and not in a silo.
Consumer brands that follow this premise know that both lift test and marketing mix modeling have to be part of their toolkit.
They’re very complementary, and a good marketing mix modeling framework should be able to integrate them and use them to train the model and inform the model’s results – even if the results change over time.
Good MMM modelers recognize that lift tests and experiments go hand-in-hand with good marketing mix modeling and can be used to reinforce each other.
But while experiments are necessary, they’re hard to do right. We just had a Recast Measurement Coffee Break with our co-founder Michael and Haus’ CEO Zach Epstein and we ran through a number of the challenges that marketing experiments come with. Here’s a summary of what they discussed:
Challenges in Marketing Experiments:
Here are some of the common mistakes Zach and Michael see when brands start running tests:
1 – Misinterpreting the results of the experiments:
People think about incremental returns and that it should reflect their CPA in some regard.
2 – Overattributing the effectiveness of media:
It’s easy to think that all of your media is incremental – it’s really not.
3 – Comparing Google’s, Meta’s, and Snap’s Conversion Lift:
Zach worked on the Conversion Lift product when he was at Google and he noticed they are not comparable across platforms.
They use different methodologies and different ways of matching individual user records.
4 – Not having a consistent form of measurement
You need a measurement system that is consistent so you can analyze and compare the results over time.
5 – Not understanding the tax of learning
Some of these mistakes above are more tactical in nature, but there’s an even bigger challenge that precedes them: not understanding the tax of learning.
This is a philosophical hurdle, and we think it’s worth zooming in on it and trying to rewire the way marketing teams see this.
The Tax of Learning from Marketing Tests:
We’re fans of really aggressive testing, but we’ve seen that some marketing teams are hesitant to run more tests or, at least, they just don’t run as many as the model would recommend.
Marketers tend to think about the cost of running a test but not the cost of NOT running them.
He shared he had seen companies lit millions of dollars on fire with inefficient media and that, if only they had been more measured and invested in testing beforehand, they wouldn’t have wasted.
The value of the insights you get from testing is very significant.
When Zach was at Google, he led the advanced attribution team and worked really closely with Netflix.
They called this “the pioneer tax.” What is the tax of learning?
You have to pay a learning tax in order to build the right things and spend your money in the most efficient way.
What is a failed vs a successful marketing experiment?
Michael added that a lot of marketers, whether they express it explicitly or not, think that if they run a test and get bad results, that’s a failed test. The value of the test is not the number of additional customers that you got or the number of customers that you potentially lost. We’re constantly working with them to reframe failed tests only if they didn’t learn anything.
I’ve seen marketers say they don’t have the budget to run a test – even when the test we propose is to reduce spending in a specific channel! And yes, even in these cases, you have an opportunity cost of missing out on additional customers, something I completely understand.
But there are better ways of structurally thinking about how much a test costs and what is the value of a test.
A good test should point you in the right direction – it does not mean that it serves as a confirmation of what the marketer wants to see. The goal of the experiment is not for the marketers to get a pat on the back – it’s to help the company become more profitable.
Let’s say you run a test on Facebook that costs $10,000. You never want to throw money around, but if we get insights and data from that $10,000 test that helps us better allocate our Facebook budget in the future, that could save us millions of dollars a year.
While it might seem frightening when you’re going to run that test, a well-run experiment will give you information that is worth the investment.
We really want marketers to think about tests that way, not in terms of what’s the cost of the test at that moment, but in considering the long-term value of the learnings.
Overcoming Challenges in Marketing Experiments
Like Zach mentioned, we’re seeing the tide start to change – and quickly. Brands are starting to design better tests, run them more often, and foster test-and-learn cultures within their marketing teams.
When we think of what the modern marketing toolkit is going to look like over the next five or ten years, experimentation is going to be a critical part of that workflow.
