Proofiness - Charles Seife [40]
The margin of error is a subtle concept, and it’s abused and misused by almost everybody who gets their hands on it—especially journalists. When the press uses a poll’s margin of error as an all-purpose gauge to declare whether the poll reflects external reality or not, they are straying from the realm of reality. And when they treat a poll with a very small margin of error as an oracle almost guaranteed to be accurate, they are making a dangerous mistake.
The margin of error only represents statistical error, the inaccuracy inherent to using a sample of a population to try to represent the whole. While that error is extremely important—it cannot be ignored—there are plenty of other errors that creep into polls that aren’t reflected in the margin of error. When polls go spectacularly wrong, the problem is almost never caused by statistical error. A more insidious kind of error—systematic error—is almost always to blame. However, systematic errors are never included in a poll’s margin of error. When journalists use the margin of error as a litmus test to figure out whether or not to believe a poll, they are completely blind to the sources of error that are most likely to render their poll meaningless. Every time a journalist cites the margin of error as a reason to believe the results of a poll, he’s doing the logical equivalent of looking only one way before crossing a two-way street. Sooner rather than later, he’ll be clobbered by a bus.
Indeed, the history of polling is filled with spectacular accidents—and it’s littered with journalistic roadkill.
By rights, it should have been one of the most accurate polls ever produced. In August 1936, the incumbent president, Democrat Franklin Delano Roosevelt, was facing a stiff challenge from Alf Landon, the Republican governor of Kansas. Literary Digest magazine began an unprecedented campaign to predict who would win the upcoming election. For two decades, the Digest had built a towering reputation for accurate polls—it was the place where political junkies could get their survey fix. Even the magazine’s advertisements were couched in the language of polls. (“Leader in every taste test, winner of every digest poll, Heinz aristocrat tomato juice is overwhelmingly elected by flavor connoisseurs everywhere!”)
According to the editors of the magazine, the secret to their success—their polls’ incredible accuracy—was their enormous sample of voters. After all, conventional wisdom was that the bigger the sample, the more accurate the poll, and each year the Digest’s sample got larger and larger. By 1936, the magazine sampled nearly one-quarter of the eligible voters in the United States. Ten million people would receive a ballot in the mail, which they could then return to be tallied by the Digest. Merely getting those ballots to their intended recipients was a truly monumental task:
This week, 500 pens scratched out more than a quarter of a million addresses a day. Every day, in a great room high above motor-ribboned Fourth Avenue, in New York, 400 workers deftly slid a million pieces of printed matter—enough to pave forty city blocks—into the addressed envelops [sic]. . . . Next week, the first answers from those ten million will begin the incoming tide of marked ballots, to be triple-checked, verified, five times cross classified and totaled. When the last figure has been totted and checked, if past experience is a criterion, the country will know to within a fraction of 1 per cent. the actual popular vote of forty millions.
Such a vast undertaking cost a “king’s ransom,” as the editors of the Digest pompously put it. “But The Digest believes that it is rendering a great public service—and when such a service can be rendered, no price is too high.”39
From the point of view of the Literary Digest’s editors, the enormous sample size more than justified the cost. If voters returned even a small proportion of the ballots, the sample would still be so large that it would reduce the poll’s margin of error to almost nothing—to within