ArticleForget Attribution – Let’s Talk Marginal Impact
Following Google’s April 6th announcement that four model types – First Click, Linear, Time Decay & Position-based – will be retired (and a significant amount of outcry from various corners of the digital advertising ecosystem), this seems like the perfect time to talk about the importance of incrementality vs. attribution (and why this isn’t that big of a deal).
Let’s start with the basics – the difference between attribution and marginal impact.
Attribution: the process for assigning credit for positive outcomes to the channel(s) which contributed to the outcome. It’s a fancy way of acknowledging that customer journeys are complex, and people generally don’t convert based on a single touchpoint. As a result, we need to divvy up the “credit” for a given outcome across all the *known* channels/tactics that contributed to it.
There are a number of flaws with attribution, but to state the obvious: (1) everyone is different – how I react to a touchpoint might be VERY different from how you do; (2) while most of us live online, not everything that happens to us happens online – we still talk to people, get the mail, etc. – all of which aren’t going to be captured by an attribution model; (3) one person = one device is rarely, if ever, true (and as I’ve learned, doubly true when you have kids!). Each of those factors, on its own, could tank the accuracy of most rule-based attribution (and a lot of data-driven attribution, too). All three? Look out. And that brings us to:
Marginal Impact (d/b/a “Incrementality”): the process of identifying which positive outcomes were caused by an activity (and thus, would not have happened but for the activity). This is the “lift” produced by marketing activities.
To simplify further: attribution is re-arranging the food on your plate to make it look like you ate a meal. Marginal Impact is eating some food. One provides an illusion of progress, the other provides progress.
With that context in mind, there are two things that become somewhat obvious:
- Attribution is – at best – an unsolvable puzzle and (at worst) a toxic black hole that hoovers up energy and resources while doing zilch to improve outcomes.
- Marginal impact is the closest thing we’re likely to get to quantifying the impact of marketing on the business – and it will never be perfect.
Historically, much of the multi-touch attribution (both in Google Analytics & via third-party tools) has relied on cookies to properly capture + assign credit for positive outcomes. But as cookies are slowly phasing out (a change catalyzed by regulation, but a long time coming either way), the basis for rule-based attribution is slowly eroding. Once Google completes the phase-out of cookies (expected in 2024, but who knows?) rule-based attribution as it is today will be a ghost of what it was just a few years ago (well, more of a ghost?).
In place of rule-based models, platforms have implemented “data-driven attribution” – dynamic models that use broad data sets, statistical inference and machine learning to model expected impacts algorithmically. There’s no two ways about it: these models are black boxes. And digital ad platforms have compelling financial incentives to ensure their models are both reasonably aligned to reality AND predisposed to favor their platform (after all, that’s how you get more advertiser dollars into your platform).
Given the current state of things (and the staggering number of lawsuits and regulatory claims against big tech), skepticism is warranted. It isn’t a stretch to say that these changes have been met with significant hostility, especially from brands that have invested heavily in building/deploying rule-based models (for better or worse). This change will be significant for those companies, but (candidly) – this is a change they should’ve made years ago.
And this is what brings me back to the point above: attribution is a shadow game. It can give hints to what’s happening in the light, but it isn’t where the action is. Just as you wouldn’t trust a shadow to show you what’s in a picture, I wouldn’t trust attribution to tell me how (or if!) to deploy marketing dollars.
We need to move marketing measurement from apportioning credit to quantifying impact. Attribution is moving food around your plate to make it look like you ate a meal. What most brands need is to consume more food (i.e. more incremental ), not make a bigger mess on their plate.
So how does all of this tie together, and (more importantly) – what do we do about it? A few points:
- The phase-out of granular, cookie-centric data will further erode the validity of rule-based attribution. This will be expedited by Google sunsetting the four models mentioned above across their Ads + Analytics products. My (unpopular) belief is that this is a good thing – because none of this was really worth doing anyway. More time and mental space freed up to do the important work of determining marginal impact and pulling the levers that impact it (see DDL Issue #1).
- Those same changes will have little-to-no impact on incremental measurement, which is much broader-based and uses statistical techniques and regression analytics to quantify impact. MMMs (and related techniques) pre-date the digital age, and they still work (though it’s nice that open-source tools like ROBYN exist to make this more accessible).
- DDA is not perfect, but I’ve found it represents what’s happening (and what I see with MMMs) as accurately, if not more accurately, than any rule-based attribution model. I don’t love black boxes, and I’m not one to blindly trust. But at this point, trusting rule-based attribution is a bigger leap of faith than DDA, due to the challenges + shortcomings noted above.
- Spend all of the time that was wasted on the attribution wars figuring out marginal impact, then using what you learned to drive improved performance across your business.
Moving from Attribution to Marginal Impact
Let’s start by stating the obvious: marginal impact is what we should all strive for. It is a bold, audacious and endlessly worthwhile pursuit, because it gets as close to true, bottom-line, dollars-in-the-bank impact as anything else marketing does.
As advertisers – whether agency or in-house – I firmly believe we should be focused on doing things that make the business better. Sometimes those are things that make marketing look good in the short term. Sometimes that’s taking an unbiased look at an initiative, admitting it failed miserably, and stopping the bleeding. The only way to do that is to obsess about the marginal impact of our marketing initiatives.
How? Three complementary methods:
Method #1: Holdout Testing
Holdout testing is straightforward: we intentionally exclude a group of people (the “control”) from a marketing campaign variable based on a property of the audience (more on that below), for a specific period of time. This should be for about 2x the median sales cycle, to ensure we’re not counting residual impact from pre-test marketing activities. The difference in conversion (or sales, or leads, or downloads, or account sign ups, or whatever outcome we’re measuring) between the control and the experimental group gives us a basis to establish the marginal impact of the marketing communications variable.
These tests is that they provide two wonderful pieces of information: (1) the marginal impact of the marketing campaign variable we’re testing on our desired outcome and (2) the baseline for what we’d get *even if we didn’t do any marketing* (more on this later).
Generally speaking, there are two broadly-acceptable methods for holdout testing:
Geographic: Exactly what it sounds like (and incredibly well established) – we exclude a few areas from marketing, while maintaining it in others. The challenge is ensuring that the locations picked are sufficiently similar to the ones in which we’re doing advertising. These tests typically include a mix of different markets/regions to exclude, so that inferences can be made across multiple vectors (for instance, incrementality of paid social might be higher in smaller markets).
Audience-Based: The alternative to geographic tests is audience split-testing. This works by targeting users on an individual level, which has become increasingly reliable on most digital ad platforms, thanks to increased emphasis on user verification and identity graphs. This avoids the challenge with geographic tests (finding the right markets) – though it can be complicated by other marketing efforts.
With either method, you’re likely to hear an objection from some people (usually 3P attribution vendors, but also agencies): you’re going to miss out on conversions from your holdout group!
Potentially, yes (if marketing has a marginal impact). But you (the savvy marketer/investor) should do the math first to determine the actual “lost” – and you’ll often see that the loss is a tiny fraction: a control group is typically 10% to 20% of your audience, and marginal impact can vary wildly, from 0% to 70%.
Putting actual numbers to this, and using a very basic lead generation example:
Assume claimed leads = 100
Control Group = 15%
Marginal Impact = 35%
Actual loss = 100*0.15*0.35 = 5 leads lost. To understand that your marketing is only 1/3rd as effective as claimed? That’s a tiny price. And – here’s the kicker – all of that “loss” might actually not be lost – it’s entirely possible that those customers will be reminded of your brand when you resume advertising to them (either geo or audience) and convert then. That 5 lost leads might net out at 2 when it’s all said and done.
Bottom line: anyone who tells you to use attribution in place of marginal impact testing probably doesn’t have your best interests in mind.
Method #2: Spend Correlation
This works best for brands with significant variable ad spend on a single platform (or at most, two platforms) – making it ideal for DTC or lead generation companies that (think) they rely heavily on paid ads.
How?
By comparing variable spend + reported conversions vs. actual sales/leads over a tight timeframe, it’s possible to determine a correlation/relationship between the investment (spend) and the outcome (sales/leads/whatever), as well as plot a decent-looking diminishing returns curve. For instance, if spend doubles but conversions remain flat, that’s a good indication that (at least a portion) of your investment might not be driving incremental outcomes. Likewise, if spend doubles and actual outcomes increase by 2.5x, that’s a good indicator there might be a multiplicative effect on spend that isn’t being properly captured by platform reporting.
This same method can also help you identify the right tCPA / tROAS in your ad account – and deal with over/under counting in Meta/Google. By comparing a source of truth (the actual new leads / new customers generated during a period) to what’s reported by the platform(s), you identify the platform coefficient, which then allows you to set better targets (and if you followed my advice in Issue #4, better cost caps or tCPAs).
(Platform Reported Leads** / New Customer Leads*) = Platform Coefficient
(Platform Reported Revenue** / New Customer Revenue*) = Platform Coefficient
*If you have multiple channels producing meaningful leads and/or revenue, that must be removed before conducting this analysis. It’s not going to be perfect, but again, it’ll provide solid, directional insight.
**I (almost) always recommend using the shortest-possible attribution window (1 Day Click on Meta + Google), to avoid situations where platforms take credit for things that happened long, long ago. There are exceptions to this rule, but not many.
Method #3: Marketing Mix Modeling (Regression Analytics)
Regression is and remains the gold standard for marginal impact analysis. Historically, that meant it was staggeringly expensive ($250k is a good benchmark for a MMM) – but today, you can get an automated MMM up and running using open-source code (like Meta’ ROBYN or Google’s Lightweight MMM) for 1/10th of the cost of a traditional MMM.
MMMs (both standard and automated) use multi-linear regression to determine the relationships between inputs (spend, channels, creative, emails, etc.) and outcomes (leads, sales, sign-ups). At first glance, this may sound relatively similar to what Method #1 + Method #2 above are doing – and there are similarities. The difference – and the rationale for the cost – is that a MMM does this at scale, using more robust methodologies, across all channels.
The outputs are thus more generalizable across the entire marketing organization, allowing a brand to compare investments in TV vs. Paid Search vs. Billboards/OOH, for instance. The advantages to using an MMM – and why I advocate building them vs. throwing money at attribution, are simple:
- Quantify Marginal Impact – MMMs are intended to quantify marginal impact of all marketing communications activities. That’s what they do. Both ROBYN and LMMM allow you to customize parameters to include different creative units (like video vs. static vs. promotional), along with different mediums (static vs. video).
- Optimize Your Marketing Investment – every marketer should obsess about the optimal allocation of capital across different channels/tactics. For a smaller brand, this likely means shifting money between 1-2 platforms (Amazon, Google, Meta); for an established brand, it could easily be 5-10 channels. MMMs facilitate this by identifying the channels/tactics that are most effective and most efficient at achieving the desired outcomes – allowing you to shift, suspend or increase investment based on actual data (vs. platform-reported attribution or gut intuition).
The drawback to an MMM is that it requires significant amounts of effort to maintain, and they tend to require some specialized expertise to configure and update. But done well – and with the proper support, a MMM can be a transformational force for your marketing efforts.
This isn’t one or the other
The final point on which I want to leave this issue is that each of these methods for determining marginal impact (“incrementality”) are complementary – you can do all of them. Doing one does not preclude you from doing the others. You can run holdout tests on platforms, and calculate platform coefficients, and invest in an MMM. They’ll all produce slightly different results (with different levels of accuracy) – but each of them will put you closer on the path to measuring impact. Each method comes with tradeoffs (ease vs. accuracy, time vs. expediency, single platform vs. whole platform) – which is why I encourage you all to test each one. See what works for your business/organization.
The sunsetting of rule-based attribution is just another step along the way to doing more good stuff (moving from moving food around the plate to actually eating). It’s not easy, but it’s worthwhile.
