Does the carry-over effect problem mean you can’t refresh your media mix model more than once a quarter?
No.
Sadly, I’ve seen a number of “experts” claiming that because marketing effects have long carry-over periods you can’t refresh a media mix model frequently. This is simply wrong and just doesn’t make any sense at all if you pause to think about it.
The claim goes something like this:
Because your marketing campaign might have effects as long as 60 or 90 days, if you are refreshing your marketing mix model every week you won’t be able to observe the full effect of marketing and will be led astray.
Sounds alarming!
But hold on! It turns out that this is going to impact our media mix model no matter how frequently we update the model. Every time you run the marketing mix model, you always have to have some “last date” in the model. That last date will always have marketing activity that is presumed to carry over into the future and will not be observed by the model because it happens after the last date.
You will have a “last date” in your model no matter whether you update the model daily, weekly, quarterly, yearly, or only once a decade.
So does the “last date” problem mean that our media mix model is unreliable no matter how often we update it?
It depends. If you use Recast, then definitely not since Recast handles the “last date” problem by always estimating the sales that will come in the future and not penalizing more recent marketing activity that hasn’t realized its full effect yet.
The way this is implemented technically is by excluding the first 60 days in the model from the Bayesian likelihood but still allowing marketing activity from the first 60 days to impact results after that period.
So, although the model is able to see the spend and the revenue from this period, the parameter estimates aren’t shaped by the fit during those first 60 days.
However, conversions after those first 60 days are impacted by spend during the first 60 days, so it’s not like we’ve just moved the problem forward.
This stops the model from over-fitting to conversions or revenue at the beginning of the time period that is influenced by things it can’t observe.
If you use other less-sophisticated MMM frameworks, then you might very well have this problem. However, there is no relationship between update frequency and the “last date” problem. If you have the problem when updating weekly, you’ll still have it when you’re updating quarterly!
