Proofiness - Charles Seife [42]
There was another more subtle form of bias at work that gently made the Literary Digest’s sample Republican-heavy. The Digest was justly proud of its impressive sample of 2.3 million ballots, but it’s important to keep in mind that the magazine had sent out ten million envelopes. This means that the vast majority of voters—more than three in four—tossed their envelope in the trash rather than sending in a response. The Literary Digest’s poll counted only the responses from recipients who cared enough to take the time to fill out the ballot and put it back in the mailbox.
People are relatively silent when they’re reasonably content, but if they’re angry they tend to shout it from a mountaintop. We don’t usually fill out a customer satisfaction survey when we’re happy with a restaurant’s service—but if the waiter was surly and the food was cold, we immediately start looking for a pencil so we can fill in the response card. When surveys and polls depend on a voluntary response, it’s almost always the case that people with strong opinions tend to respond much more often than those who don’t have strong opinions. This introduces a bias; the poll disproportionately reflects extreme opinions at the expense of moderate ones.
In the 1936 election, the Republicans were out of power. Discontented with Roosevelt’s policies and frustrated at their inability to change them, the typical Republican voter was unhappy with the government. A vote for Alf Landon was a vote for change, a vote of dissent. Democrats, on the other hand, tended to be content with the government. A vote for Roosevelt was a vote for staying the course, for the status quo. When the Literary Digest’s envelopes arrived in the mail, the angry, discontented Landon voters were much more likely to send in a response than the content Roosevelt ones. As a result, the flood of cards coming back were disproportionately Republican.
Both of the Literary Digest’s mistakes had the same effect. Mailing to mostly well-to-do households biased the sample toward Landon voters, as did the “volunteer bias” caused by the reliance upon people’s willingness to fill out a ballot and mail it back. With this particular poll, these two errors were enormous, throwing the result off by tens of percentage points, even though the margin of error was incredibly tiny. What doomed the Literary Digest poll was systematic error, not statistical error.
Systematic errors are much more dangerous than statistical errors. They can be extremely subtle—they often manage to be completely invisible until they smack you upside the head. They can be difficult (and expensive) to avoid. And they come in a variety of forms, each one deadly in its own way.
George Gallup made his name, in part, by predicting the Literary Digest failure. His polls were sophisticated: by “scientifically” choosing samples carefully to make sure that they were representative of the population, Gallup produced some of the most accurate polls of his day, even though he used much smaller sample sizes than some of his more money-flush peers. He gained a reputation for reliability—and he helped renew journalists’ faith in polling even before the embers of the Literary Digest conflagration had cooled. But even Gallup’s “scientific” methods were subject to systematic errors.
The “Dewey Defeats Truman” headline was the consequence of Gallup’s most famous failure. In the weeks before the 1948 election, his polls showed that Dewey was way ahead—so much so that his employees