Occasionally, prospective customers react with shock when we tell them that we don’t “control for the COVID pandemic”. “The pandemic really impacted our business, so how can your model be accurate if you don’t control for Covid?” they ask.
The answer is simple: Recast models changes in the macro-environment, but doesn’t control for them. This is actually a very subtle but important distinction that we want to try to lay out here. We’ve touched on this before, when we explained how most people model seasonality the wrong way. In this post, I’ll try to give both intuitive explanations for the modeling decisions, as well as slightly more rigorous mathematical arguments for this approach.
At Recast, we work with businesses in many different verticals. Some were positively impacted by changes to consumer behavior associated with the pandemic response, and some were negatively impacted.
Model Covid, don’t “control for” it
The most important idea here is that at Recast we believe that Covid did not have a purely additive impact on a company’s business completely independent from their marketing. We believe that Covid changed the effectiveness of marketing itself, and didn’t just impact baseline or organic sales.
First, let’s get some intuition for what we mean by this. Imagine you’re a digital platform that helps people book reservations at restaurants. This sort of business was strongly negatively impacted by the Covid pandemic since consumers dramatically reduced the amount of restaurant reservations they were making in 2020 and 2021. If you are running the Facebook campaigns for this platform, would you expect your ROI to change between January and April 2021? I think most marketers would say that the answer here is “obviously yes”. You could spend a hundred million dollars on marketing, and it wouldn’t matter if people are afraid to leave their houses.
Even during lighter lockdown scenarios, for every dollar you’re investing in marketing, you are getting fewer additional conversions and less revenue, because consumer demand has changed and your offer to consumers is less compelling.The Covid pandemic impacted your organic / baseline sales, of course. But it has also made all of your marketing efforts less efficient. When we say that Recast models the impact of Covid, that’s what we mean: Recast’s estimates of channel performance change over time, and during periods where your marketing is less effective, Recast will reflect that.
So what is Recast doing differently? Recast is different because it estimates how the ROI for each marketing channel changes over time. So during the pandemic, to the extent that Facebook marketing performance gets worse, Recast will show a lower ROI during that period. The model doesn’t need to “control for” the pandemic, because the model will automatically incorporate the effects of the pandemic into the estimates of channel efficiency.
In this case a traditional MMM might update quarterly (or even annually!), so you get the same answer for “what’s the impact of marketing” through the whole quarter, and obviously bad assumption in times of huge consumer change like during a pandemic. Empirically, we’ve seen this with many clients – we see in real time (and looking back historically) that channel performance changed in response to the pandemic without the model knowing that there was a pandemic happening at all. This happens automatically within Recast models, so you can respond in real time to take advantage of peak demand, or scale back in response to sudden declines, without updating your model or even knowing what caused the shift.
This is exactly what we want: it keeps our model estimates actionable (we’re not estimating the effectiveness of marketing spend in an imaginary world where there’s no pandemic) and the model will update its estimates in real time: as soon as conditions on the ground change, the Recast model will respond to improved (or worsened) performance without having to make a prediction about when the pandemic will ease (or come back). We don’t have to directly estimate the impact of COVID, or any other small or large external factor. It just works.
What does it mean to “control for” the pandemic?
When other modelers talk about “controlling for” Covid, they mean something very specific. What they mean is that their model assumes that the impact from Covid is an independent effect that can simply be added to the other effects that make up the model’s prediction. This type of model would say that Covid had a negative impact on your business, but that your Facebook performance stayed the same before and after Covid. Most marketers would say that this is not a very good assumption!
Let’s look at the math. In an additive model (like the ones used by Facebook’s Robyn and Google’s Lightweight MMM, and the vast majority of MMM vendors), we are working with a statistical model that’s structured like this:
The way these models work, the right side of the equation must add up to the left side of the equation. That is to say, all of the effects being modeled must add up to the total revenue of the business. So, including additional control variables (adding things to the right side of the equation) will change both the estimates of the other variables as well as their interpretation.
Let’s imagine that we have revenues of $100 and an initial model estimate that looks like this:
Now, let’s imagine that we control for Covid, where covid had a negative $50 impact on the overall revenues of the business. Now our equation looks like this:
Here we see that our equation no longer adds up correctly. This means that our ROI estimates must adjust in order to account for Covid. And which direction are they going to adjust? Well they have to go up. That is, controlling for Covid in this scenario increases our ROI estimates for our marketing channels. Now they might look something like this:
By controlling for Covid, we somehow increased the ROI of our marketing channels. It also changes the interpretation of those variables. The interpretation of the coefficients for our marketing spend channels is no longer “what is the impact of adding an additional dollar to facebook or google?” but now rather “what is the impact of adding an additional dollar to facebook or google if there was no effect of the pandemic.” This is a very important difference!
While it might be interesting to know the answer to that question about the hypothetical world where the pandemic didn’t exist, it’s actually not actionable. As a marketer, you are running this marketing campaign in this world where the pandemic does exist, and the hypothetical ROI that you would get if the pandemic didn’t exist might look better when you give your CEO the readout on marketing performance, but it will lead you astray if you make your buying decisions based on that “fake world” ROI.
Additionally, if you take these results to your finance team and you tell them that your model shows that your ROI on facebook spend is 3x and your ROI on google spend is 1.75, they’re going to laugh you out of the room. They’ll very quickly note that 20*3 + 40*1.75 = 130, which is more than the total revenue you got over this time period. Something in this model doesn’t match the real world your team is actually operating in! If you want to avoid this scenario, you need to stop controlling for external factors, and instead think about how to model them.
- Revenue here could be switched to “new customer conversions” or “app installs” without loss of generality.
- In this blog post I’m going to review a hypothetical business that was negatively impacted by the pandemic, but all of the points I make will generalize to the opposite case where all of the effects are reversed.
