Last updated: March 11, 2024
Google’s Meridian MMM platform was just announced last week and is currently in closed beta and so here at Recast we have not had access to the actual modeling package. As we get more updates on the package (including access to source code) we will update this blog post.
Google recently announced the launch of their new open source media mix modeling (MMM) software called Meridian. This package will eventually replace Google’s existing MMM modeling software LightweightMMM.
Based on the documentation, there will be a lot to like in this package for those experimenting with media mix models. However, it also appears that some fundamental limitations of LightweightMMM are being carried forward into Meridian. In this blog post, we’ll walk through the key features of Meridian as well as some of its limitations.
Meridian’s Key Features:
- Fully bayesian (like LightweightMMM)
- Geography-based hierarchical model (like LightweightMMM)
- Time-varying intercept
- Can incorporate search volume
- Can incorporate reach and frequency measures
- Lots of good documentation on causal inference in an MMM context
Meridian’s Limitations:
- No time-varying covariates
- Unclear how to incorporate experimental results
- No handling of upper funnel vs. lower funnel channels
- No constrained optimization
- Limited features around model validation and stability evaluation
We’ll talk through each of these points one by one in this blog post.
The Features
Overall, Meridian is a big step forward in terms of the open source MMM packages that are available to marketing scientists. There’s a lot to like here.
Fully Bayesian
Meridian, like its predecessor the currently-available LightweightMMM, is a fully Bayesian statistical model estimated using tensor flow. At Recast, we aren’t zealots but we do believe that Bayesian statistical models are the right approach to take for estimating complex models on observational data as we do with MMM. The key advantages of a bayesian approach are:
- It allows for an extremely flexible model specification without taking shortcuts
- It correctly propagates uncertainty through the entire model and all parameter estimates and forecasts
- It allows all parameters (including adstock and saturation) to be estimated with uncertainty instead of simply “chosen” by the modeler
The Bayesian approach allows for the modeler to be crystal clear about the assumptions they’re making in the context of the model.
Geographic Hierarchical Model
Like LightweightMMM, Meridian allows modelers to estimate a geographical hierarchical model. That means that the modeling framework can take advantage of observing more variation in marketing activity and business performance (since each geography is a little bit different) often yielding more precise parameter estimates as well as producing results at a geographic level.
The downside of taking this approach is simply one of engineering: managing all of the data sets to feed into a geographic hierarchical model can be a real pain without dedicated data engineers contributing to the task.
Time-varying Intercept
This is a big improvement over what is currently available in LightweightMMM which assumes that the intercept (representing something like “organic sales”) is static over the entire time period being modeled. Meridian will have a feature that allows the analyst to choose how many knots (something like “change points”) to allow in the intercept.
This theoretically allows for a more flexible way to capture business growth or seasonality in the model, though it’s implemented very differently from how it is in Recast which uses a Gaussian Process under the hood.
Search Volume
One of the innovations we’re most excited about in this package is the ability to include search volume in the MMM to account for endogeneity related to paid search marketing. In 2018, some researchers at Google published a great paper on this problem and proposed some solutions (see link here) so we’re hopeful that the Meridian implementation is along those lines.
While there’s some discussion in the package of the modeling implications for search marketing, it’s very light on details so we are still waiting to see the code in order to better understand exactly how the model is structured.
Reach and Frequency Measures
Media buyers purchasing video ads (Linear TV, Streaming TV, YouTube, etc.) often spend a lot of time thinking about reach and frequency and design their marketing budgets around certain reach and frequency targets.
The Meridian MMM model allows for the incorporation of reach and frequency data into the MMM itself which is an interesting innovation.
At Recast, we aren’t sure that this approach is actually better than including a variable like spend since reach and frequency are often estimates from a black-box model and are not necessarily consistent over time or a good representation of advertising quality. However, we’re very excited to see how this ends up working in practice.
Documentation on Causal Inference in MMM
One of my personal favorite things about this package is all of the great discussion they include on the advantages of the Bayesian approach and the limitations of causal inference in the context of MMM. They have a section on causal estimands, their causal graph, and using MMM as a causal inference methodology.
While we don’t necessarily agree with all of their assumptions, we do applaud the transparency and clear discussion of limitations.
The Limitations
While Meridian appears to be a big step forward in terms of open source MMM packages, there’s still a number of limitations that users should be aware of before using any results from the package to make actual marketing budget decisions.
No Time-varying Covariates
The most important limitation of Meridian is that it assumes that marketing performance doesn’t change for any channel over the time period being modeled. That is, it assumes that a brand’s performance on Meta is the same last week as it was last year as it was the year before that.
Unfortunately, this assumption doesn’t match what any marketer or marketing scientist believes about how marketing works in the real world. Channel performance may change for any number of reasons including creative, targeting strategy, competitor activity, or even just the platform itself (e.g., the whim of Zuckerberg). This is a very important limitation and dramatically increases the risk that the model is misspecified and will lead to biased (i.e., “wrong”) results.
Unclear How to Incorporate Experimental Results
MMMs and experiments go hand-in-hand and while Meridian’s documentation mentions using experiments to calibrate the model multiple times, it’s unclear how that should happen practically other than via the prior-setting process.
The main problem is that since Meridian doesn’t allow for time-varying covariates (i.e., that a marketing channel’s performance can change over time) it isn’t clear how you would incorporate the results of multiple different experiments that have different results. At Recast, we recommend that companies test their largest channels multiple times a year and we know from experience that a channel’s performance can change dramatically over time. So a good MMM should be able to make sense of the results of multiple different experiments for a single channel where the results will be different over time.
No Handling of Upper Funnel vs Lower Funnel Channels
Meridian documentation mentions that lower funnel channels like search can be handled by controlling for search volume but 1) that implementation is a bit light on details and 2) it doesn’t mention anything about other lower-funnel channels (like affiliates or email marketing). Being able to understand the interaction between upper-funnel and lower-funnel channels is important for an MMM but Meridian (as of yet) doesn’t seem to have this feature.
No Constrained Optimization
The optimization engine that comes out of the box with Meridian allows for the optimization of a single budget or to try to hit a certain ROI or KPI goal, though the docs don’t indicate that there’s a way to constrain that optimization to be something reasonable. Constrained optimization is required in order to generate realistic budgets that a business can actually use, otherwise these types of MMM models often recommend budgets that are completely unworkable from a business perspective (like putting all spend into one single channel).
Limited Features Around Model Validation and Stability Evaluation
At Recast, we often say that running an MMM is trivially easy, and the hard part is knowing if your answers are right or wrong. We recommend a thorough model validation process that includes parameter recovery exercises, robustness checks, as well as ongoing holdout forecast accuracy. All of these efforts require a lot of engineering work on top of the model itself and don’t seem to be provided easily in the package.
Final Thoughts
There is a lot to like about Google’s Meridian MMM package, including its flexible model specification, propagation of uncertainty through the entire model and estimation of parameters that allows for uncertainty.
However, practitioners should be mindful of some fundamental limitations carried over from LightweightMMM including Meridian’s lack of time-varying covariates, unclear handling of upper funnel vs. lower funnel channels, and more.
We’ll be following Meridian closely and will update this blog post as we receive updates and get access to source code.
Note: If you like these breakdowns, check out our write ups on LightweightMMM and Facebook Robyn.
