Many people want to know what the long-term effects of their marketing are. The theory is that advertising today may boost brand awareness which might have long-term positive impacts for the business. Often, they want to know how Recast thinks about these long-term impacts.
Unfortunately, we believe that it is practically impossible for an econometric model (any econometric model) to estimate such effects except for very specific circumstances. The logic is as follows: Econometric models rely on variation. Brand awareness metrics have very little variation and are noisy.
These two factors combined mean that there is very little underlying signal in the data and an econometrician might say that the model is “unidentifiable”. Unfortunately, this doesn’t actually prevent an analyst from including a term in the model for “long term brand effects”, but what happens is that the analyst can effectively make that term say anything at all. Since there’s no underlying signal in the data, it’s trivially easy to specify a model that shows a high or low brand effect.
Let’s walk through this in a bit more detail.
Variation in Econometric Models
Econometric models rely on variation in the inputs and outputs in order to “identify” a parameter. If we imagine a world where marketing spend and sales were perfectly flat for all of history, there would be no way for a model to “identify” the relationship between changes in spend and changes in sales, since there are no changes!
This is a fundamental limitation of the universe: all statistical models have this limitation.
Brand Awareness
Brand awareness metrics tend to have very little movement. Many brands that I’ve worked with have had their brand awareness survey yield results within a few percentage points of each other over years. Regardless of how much they spend, their brand awareness just doesn’t change much.
There are exceptions to this rule of course: it’s easy to think of Airbnb or Uber as examples where their brand awareness changed dramatically over the course of a few years. However, Uber and Airbnb both probably have very flat brand awareness now – after those few years where they grew a huge amount in popularity, I’d wager that their brand awareness metrics are extremely flat. And if the metric is flat, it means there’s no signal for an econometric model.
Beyond being fairly flat metrics in general, brand awareness metrics tend to be based on survey results which are very noisy due to measurement error. Marketers sometimes ignore this, but any good brand awareness survey vendor should communicate to you both the average metrics as well as the margin of error. The margin of error tells us the uncertainty in the results. So we can imagine data like this from our survey vendor:
- October Unaided Awareness: 35% +/- 5%
- November Unaided Awareness: 37% +/- 5%
- December Unaided Awareness: 39% +/- 5%
Is the unaided awareness trending up? No. Or, it might be, but these survey results don’t tell us that since all of these results are within the margin of error of each other.
Simply including the mean responses in a regression for an MMM without incorporating the margin of error is a huge modeling mistake, which is unfortunately common with many econometrics vendors.
Brand Awareness vs “Baseline demand”
So, since brand awareness measures are generally both flat and noisy, it’s very difficult to use that variable to provide signal in an econometric model.
Econometric marketing models generally have a term in the model for “base demand”. This is estimated statistically using an “intercept” in a regression model. Effectively it’s the term that captures all of the sales that aren’t driven by the other effects in the model.
Unfortunately, when we include a variable like “brand awareness” in the model, the model can’t differentiate between what is “base demand” (the intercept) and what is coming from brand awareness. By making very small configuration changes, the analyst can often make the model say whatever they want since the model will randomly1Not exactly randomly, but for the purposes of marketers reading this it’s effectively random. allocate effect between the base demand and the brand awareness terms.
Conclusion
Unfortunately, this leaves us in a pretty tough spot. As much as we’d like to be able to understand the impacts of marketing on brand awareness, and brand awareness on long-term sales trends, in most cases2If you are on the way to being the next Uber and your brand awareness is on a steep upward trajectory, let’s talk! there simply isn’t any way to measure those effects accurately.
