For many businesses, email programs are a huge revenue generator. For every $1 marketers spend on email marketing, they receive $36 in return, according to 2,000 marketers who responded to Litmus’ 2020 State of Email Survey. Emails play the important role of nursing customers along their buying journey, drive retention, and cross-sell and up-sell customers to new product classes, so it’s no surprise that marketers value their contribution
Since email is an important part of their business, people often ask us “what’s the ROI?”, expecting to see it nicely lined up in their list of channels in Recast’s MMM insights dashboard. We hate to be the bearer of bad news, but modeling email isn’t that simple. In general, email marketing functions differently than most paid marketing channels. Paid marketing channels drive new customers to the website, where they may or may not sign up for an email list (to get a discount, for example) and may or may not make a purchase (at which point they’ll be added to an email list and targeted for cross-sell, up-sell, and retention efforts).
Email marketing is by definition a bottom-of-funnel channel. It can help convert a customer once they’re already familiar with a brand, and it can help improve their LTV, but it cannot drive awareness or discovery. Investment in email can only generate that remarkable 36 to 1 return, if you’ve invested in building that email list in the first place.
To some extent, I often think about investing in an email marketing program as being more like investing in a high-quality e-commerce site than investing in paid marketing. A good email program will improve the returns to your marketing efforts but is not a substitute for them. Just like investing in improving product recommendations or reducing friction in the checkout flow complements your marketing campaigns but can’t replace them.
Different Types of Emails
So, given all this complexity, how do we handle email at Recast? Depending on your strategy, we will collaborate with your team to include email in a way that makes sense to your business. Email is likely to impact customer behavior and how the model results can inform your most pressing decisions related to email, so we take it into account, even if we don’t control for it. Here are our approaches to the common components of an email program, based on real client experiences and the solutions we’ve delivered.
- Weekly newsletter-type emails – There is often little measurable variation except the number of subscribers. Recast relies on data variation to define the impact of channels, so Recast (and any MMM, not just Recast) is not likely to be able to tease apart this impact from a) the day of the week impact b) the number of subscribers you have, which is a dependent rather than independent variable.
- Email blasts / bulk sends to owned audiences: We include promotions/events in our model, and often email is a part of getting a promotion attention. We can test layering an additional variable to tease apart the email vs other channel impact, as it relates to a specific event/promotion.
- Email blasts / bulk sends to a partner audience: We would definitely want to include this variable in the model. Depending on whether and how this activity is paid for, we may recommend including spend or modeling a non-spend spike.
- Automated Nurture – New Customers: A great email strategy (and push, in-app, SMS, for that matter) is part of your product and the user experience of your product. When significant changes are made to a nurture strategy, we model them in the way we model product releases. An improved product increases the performance of all top and bottom of funnel activity because it makes it more likely that the new user will convert, or it reduces the customer time to convert. We recommend continually improving products, including these communication components, and then clients can infer how that contributed to the changes in media ROIs.
- Automated Winback and Retention: These can be great for revenue, but we don’t want to directly include them in the model or calculate their ROI because it is difficult to know what to do with that information. For example, maybe we usually send 100 winback emails per week and win back 5 of those customers. Last week, we sent only 50 and won back none. Last week, our ROI was… well, that depends on how we calculate spend… but let’s say it was high and now it’s low. What we would do with that information on any other channel is simple: either fix it or spend less. But for email, what we do depends. The ROI may be low because the cohort of users came from a better channel. Or a promotion last week kept the other 4, who would usually churn by now, from qualifying for winback. Not knowing the why and just seeing the ROI may lead some marketers to 1) Invest more in this channel up to a level where we can send 100 winback emails, because when we were at that threshold we had a better ROI 2) Invest less in this channel because it is performing poorly. Neither makes sense.
When you think about your retention and win back strategy as part of your product, and make decisions about investing in those channels in the way you make product decisions, your business is generally better off.
That seems complicated. Can’t we keep it simple and just put total emails into the model?
When it comes to the modeling decision, one thing many people don’t realize (unfortunately, this includes many professional modelers) is that adding additional variables to a model changes the interpretation of all of the other variables in the model. This is critically important in media mix modeling where, if you want to make the results actionable, you need to be laser-focused on how the coefficients of your model can (and should) be interpreted. The approach of just throwing in more control variables until the model has a high R-squared metric is a terrible approach to model building and will lead to nonsensical or non-usable results. (If you’re interested in going deep on why this is a bad approach, you can start here and here.)
Before we continue, I want to note that this is true of all regression-based models, not just Recast. Everything we’re going to talk about in this blog post applies to Facebook’s Robyn, Google’s Lightweight MMM, Recast, and every other MMM model that I’ve ever encountered. This is a fundamental limitation of statistical analysis, not an issue with any particular tool.
Why does this matter?
This matters because with an MMM we want to estimate the incrementality of our marketing investment. Explicitly, the question we want to answer is “If I were to put an additional $100 of investment into this marketing channel, how much additional revenue would that drive?”. This framing is important because it includes the effect of downstream email marketing. The question is not “how much additional revenue would that drive if we didn’t have an abandon-cart email flow?” or “how much additional revenue would that drive if we hadn’t improved our e-commerce site?” or any other version of that question. We want to know how much additional revenue that marketing investment will drive given all of the downstream tools that exist for converting the interest generated by the investment into revenue.
Let’s take a look at the regression formula that lies at the heart of all MMM models:
Revenue = baseline_sales + FB_spend*FB_ROI + google_spend*google_ROI
The idea here is that we decompose revenue into different parts. The “baseline” sales (sometimes called the “intercept”) and the pieces of revenue driven by our different marketing investments. The interpretation of the “FB_ROI” and the “google_ROI” variables are as follows:
How much additional revenue does an extra dollar of spend invested in this marketing channel drive, holding constant the investment levels in the other marketing channels?
This is exactly what we want! Because we want to know what’s the incremental value of that investment, which lines up perfectly with the interpretation of the ROI estimates in this formulation.
However, if we include emails being sent in this equation
revenue = baseline_sales + FB_spend*FB_ROI + google_spend*google_ROI + emails_sent * revenue_per_email
then things get really tricky. Because now we interpret the “FB_ROI” and the “googlel_ROI” variables as follows
How much additional revenue does an extra dollar of spend invested in this marketing channel drive, holding constant the investment levels in the other marketing channels and without being able to send additional emails
That is to say, now we’re estimating the impact of our marketing channels in a world where we can’t send them additional emails. In my opinion, this doesn’t make sense! It’s not the right framework for making marketing spend investment decisions because we are planning on sending emails to customers whom we advertise to!
So, by “controlling for” emails sent in our model, we’ve ruined our ability to interpret our estimated coefficients for our marketing channels as estimates of “incrementality”.
Just because the email team sits under the “marketing” banner doesn’t mean that controlling for emails makes sense in the context of an MMM. We believe that the email team should be treated more like a digital product or web team in this context: the work that they do should make all of our marketing efforts more efficient, but should be evaluated separately from an MMM model.
