At Recast, we’ve built our media mix modeling (MMM) platform based on Bayesian methodology because we believe it provides the most powerful and flexible way to understand marketing performance. It lets us incorporate prior knowledge, account for uncertainty, and generate forward-looking predictions – which is what you need from your MMM.
However, estimating a Bayesian MMM does have some challenges, and there are three that are most common:
- Tree Depth Saturation – When the sampler doesn’t fully explore the parameter space.
- Divergences – When the numerical integration process breaks down.
- Multimodality – When the model finds multiple conflicting solutions.
Each of these issues can impact the accuracy and stability of your MMM, and that’s why we’ve spent a lot of time making sure that our models solve for them. Let’s break them down one by one.
1. Tree Depth Saturation: When the Sampler Stops Too Soon
What It Is
To estimate a Bayesian model, we need to explore the posterior distribution—the range of possible parameter values that fit the data. The No-U-Turn Sampler (NUTS) does this by building a tree of potential solutions, expanding deeper and deeper into the parameter space.
However, tree depth saturation happens when the sampler hits a predefined depth limit before it has fully explored the posterior. This means it may stop before reaching the most relevant parts of the distribution.
Why It’s a Problem
Tree depth saturation can truncate the model’s understanding of uncertainty, meaning:
- The model may only explore a subset of possible parameter values.
- The final estimates might be biased toward the parts of the posterior that were explored.
- The model might fail to capture the full range of potential outcomes, leading to overconfidence in certain conclusions.
If you have tree depth saturation, it means that you’re not necessarily getting the full picture. The sampler might only be focused on one possible subset of the answers rather than fully exploring all possible solutions.
How Recast Handles It
At Recast, we actively monitor tree depth to ensure our samplers are covering the full posterior distribution. Some ways we do this:
- Adjusting tree depth parameters so the sampler can explore deeper if needed.
- Running diagnostic tests to identify when the sampler isn’t fully traversing the posterior.
- Reparameterizing the model to improve exploration efficiency and avoid unnecessary depth limitations.
2. Divergences: When the Model Can’t Explore Certain Areas
What It Is
Divergences happen when the numerical integration process used in HMC fails. The sampler is supposed to move smoothly through the posterior, but sometimes, it comes across regions of high curvature—like a steep valley or a sharp cliff—where it can’t take stable steps.
This typically happens when:
- The posterior distribution has highly complex geometry, like extreme curvatures or narrow bottlenecks.
- The model’s parameters are strongly correlated, which makes it difficult for the sampler to move through the space efficiently.
Why It’s a Problem
Divergences indicate that the model isn’t properly exploring certain parts of the parameter space, which can lead to:
- Under-explored regions of the posterior, which means some valid parameter values are missed.
- Biased parameter estimates since the sampler might avoid difficult regions.
- An incomplete view of uncertainty makes the model’s confidence in its predictions unreliable.
Divergences are like a flashing red warning light—they mean the sampler is struggling to explore certain areas of the model, and that’s a big problem.
How Recast Handles It
Recast tackles divergences using multiple strategies:
- Reparameterizing the model to smooth out problematic areas in the posterior.
- Tuning step sizes in the sampler to ensure it can handle sharp curvatures.
- Using regular diagnostics to catch and fix divergence issues before finalizing results.
3. Multimodality: When the Model Finds Multiple “Right” Answers
What It Is
A multimodal posterior has multiple peaks, meaning the model sees two or more distinct solutions as equally plausible.
Imagine a landscape with two mountains. Depending on where the sampler starts, it might land on one peak or the other, even though they lead to very different conclusions.
Why It’s a Problem
- Inconsistent results: one week, the model might estimate that a channel’s ROI is 5x; the next week, it might estimate 0.5x—simply because it landed on a different mode.
- Unreliable predictions: if the model jumps between two conflicting solutions, its recommendations can be unstable.
- Hard to interpret: decision-makers will struggle to trust insights if they keep changing.
How Recast Handles It
To solve for multimodality, we:
- Run multiple sampling chains from different starting points to check for alternate modes.
- Analyze convergence diagnostics to make sure the model isn’t stuck in one mode.
- Refine priors and constraints to prevent unrealistic modes from forming.
4. How Recast Ensures a Robust Bayesian MMM
Tree depth saturation, divergences, and multimodality are three of the biggest challenges that come with estimating Bayesian MMM, and they come down to rigor. Your MMM team –whether a vendor like Recast or in-house– needs to be obsessed with making the model accurate and trustworthy.
In summary, effectively managing tree depth saturation, divergences, and multimodality is key to reliable Bayesian MMM. Rigorous diagnostics, precise tuning, and multiple sampling runs ensure that the full uncertainty is captured, leading to stable and trustworthy insights for better marketing decisions.
