Everything Is Obvious_ _Once You Know the Answer - Duncan J. Watts [88]
In theory, of course, everyone “knows” that correlation and causation are different, but it’s so easy to get the two mixed up in practice that we do it all the time. If we go on a diet and then subsequently lose weight, it’s all too tempting to conclude that the diet caused the weight loss. Yet often when people go on diets, they change other aspects of their lives as well—like exercising more or sleeping more or simply paying more attention to what they’re eating. Any of these other changes, or more likely some combination of them, could be just as responsible for the weight loss as the particular choice of diet. But because it is the diet they are focused on, not these other changes, it is the diet to which they attribute the effect. Likewise, every ad campaign takes place in a world where lots of other factors are changing as well. Advertisers, for example, often set their budgets for the upcoming year as a function of their anticipated sales volume, or increase their spending during peak shopping periods like the holidays. Both these strategies will have the effect that sales and advertising will tend to be correlated whether or not the advertising is causing anything at all. But as with the diet, it is the advertising effort on which the business focuses its attention; thus if sales or some other metric of interest subsequently increases, it’s tempting to conclude that it was the advertising, and not something else, that caused the increase.17
Differentiating correlation from causation can be extremely tricky in general. But one simple solution, at least in principle, is to run an experiment in which the “treatment”—whether the diet or the ad campaign—is applied in some cases and not in others. If the effect of interest (weight loss, increased sales, etc.) happens significantly more in the presence of the treatment than it does in the “control” group, we can conclude that it is in fact causing the effect. If it doesn’t, we can’t. In medical science, remember, a drug can be approved by the FDA only after it has been subjected to field studies in which some people are randomly assigned to receive the drug while others are randomly assigned to receive either nothing or a placebo. Only if people taking the drug get better more frequently than people who don’t take the drug is the drug company allowed to claim that it works.
Precisely the same reasoning ought to apply in advertising. Without experiments, it’s actually close to impossible to ascertain cause and effect, and therefore to measure the real return on investment of an advertising campaign. Let’s say, for example, that a new product launch is accompanied by an advertising campaign, and the product sells like hotcakes. Clearly one could compute a return on investment based on how much was spent on the campaign and how much sales were generated, and that’s generally what advertisers do. But what if the item was simply a great product that would have sold just as well anyway, even with no advertising campaign at all? Then clearly that money was wasted. Alternatively, what if a different campaign would have generated twice as many sales for the same cost? Once again, in a relative sense the campaign generated a poor return on investment, even though it “worked.”18
Without experiments, moreover, it’s extremely difficult to measure how much of the apparent effect of an ad was due simply to the predisposition of the person viewing it. It is often noted, for example, that search ads—the sponsored links you see on the right-hand side of a search results page—perform much better than display ads that appear on most other Web pages. But why is that? A big part of the reason is that which sponsored links you see depends very heavily on what you just searched for. People searching for “Visa card” are very likely to see ads for credit card vendors, while people searching for “Botox treatments” are likely to see ads for dermatologists. But these people