“I ran an experiment showing positive lift but didn’t see the results in the bottom line.” Sounds familiar? I think we’ve all had this experience: we set up a nice, clean A/B test to check the value of a feature or a creative. We get the results back: 5% lift, …
The Hidden Pitfall in Marketing Experiments: Understanding Uncertainty Read More »
Does the carry-over effect problem mean you can’t refresh your media mix model more than once a quarter? No. Sadly, I’ve seen a number of “experts” claiming that because marketing effects have long carry-over periods you can’t refresh a media mix model frequently. This is simply wrong and just doesn’t …
Carry-Over Effects: Why They Don’t Limit Media Mix Model Updates Read More »
Is your incrementality testing partner hiding something from you? We’ve seen it too many times: brands come to us very proud of running an incrementality study and they share the results with us and say something like: “Look, we ran an incrementality study and there was a 3% lift and …
The Hidden Truth Behind Incrementality Testing Results Read More »
“There is nothing so stable as change” – Bob Dylan Why Measure Model Stability? Media mix models (MMMs) need to be actionable for marketers. If an MMM produces wild swings in output every time it receives new information, it indicates that there is something wrong with the model and it …
MMMs are powerful models and that means they’re subject to what modelers call “over-fitting”. The idea is that you can build a model that fits really, really well to the data that the model is trained on, but that hasn’t found the actual underlying causal relationships in the data. Over-fitting …
Understanding Overfitting in Media Mix Modeling: Risks, Metrics, and Validation Read More »
Correlation vs causation is a huge marketing measurement problem that traditional methods often fail to account for and can lead to biased conclusions and misguided budget allocation. This is where Directed Acyclic Graphs (DAGs) come into play. DAGs offer a powerful, visual approach to mapping out and clarifying causal relationships …
How do Causal Directed Acyclic Graphs (DAGs) work in Marketing Mix Modeling: Read More »
Media mix modeling at the enterprise level can be quite complex but, when done well, has a huge impact across the whole organization. Here are 5 lessons we’ve learned to help make MMM work at the enterprise: 1. At the enterprise level, you need one central point of contact who …
5 Key Lessons for Successful Enterprise-Level Media Mix Modeling Read More »
Over the past few decades, the scientific community has grappled with what is now known as the replication crisis—a widespread recognition that many significant results published in academic research cannot be replicated under rigorous testing. This crisis has shown critical flaws in traditional statistical methods, particularly the over-reliance on P-values, …
The Replication Crisis: Challenges and Lessons for Modern Statistical Practices Read More »
Is AI revolutionizing media mix modeling? People have been asking about how AI is changing media mix modeling, and the answer really depends on exactly how you define “AI”. The Role of Large Language Models in MMM Large language models, including ChatGPT, don’t really have a lot of applications to …
Can you trust your media mix model? That is an obvious but not asked enough question that everyone using MMM should consider – in-house, with a vendor, it doesn’t matter. As you try and answer it, you’ll run into overfitting, which is one of the biggest problems in MMM. Overfitting …
Evaluating Media Mix Model Accuracy: The CRPS Score in Backtesting Read More »
