Proofiness - Charles Seife [45]
It’s not just Americans who lie to exit pollsters. In 2006, political observers were stunned when the Palestinian elections ended in a tremendous upset. Polls—including exit polls—had predicted that the Fatah party would beat its rival, Hamas, by about six percentage points. Instead, Hamas won by a few points. Again, the mistake dwarfed the polls’ margins of error. In fact, the reason for the mistake was that Hamas supporters pulled an anti-Buchanan. They tried to make Hamas appear less popular than it really was, by either lying to pollsters or refusing to talk to them. Why? One of the poll organizers told the Wall Street Journal that “Hamas sought to influence the outcome of the exit polls in the hope that the exit poll results would not alarm Fatah armed groups for fear [Fatah] would seek to harm the electoral process (for example, by burning ballot boxes as happened in previous elections at the local level).” If the government burns your ballot boxes when there’s too much support for your party, it makes perfect sense to lie in a poll. Unfortunately, poll manipulation is all too common. There are plenty of other actors who try to manipulate the outcome of polls. The worst are the pollsters themselves.
Since the dawn of polling, researchers have known that the outcome of a poll can be affected quite dramatically by the wording of the questions in the survey. The choice of words in even the most neutral-sounding phrase sometimes subtly steers a subject to answer in a particular way. For example, in the early 1990s, Gallup (teaming up with CNN and USA Today) asked Americans whether they supported bombing Serbian forces in Bosnia. The response was extremely negative: 55 percent against to 35 percent in favor. The very same day, ABC News asked a similar question in a different poll, and the answer this time was overwhelmingly positive: 65 percent supported airstrikes, while 32 percent opposed them. How could two polls give such amazingly different results? The mismatch was because of a subtle difference in the way the questions were worded. In the Gallup poll, the United States was doing the airstrikes. In the ABC poll, the bombings were carried out by the “United States, along with its allies in Europe.” Such a tiny change in wording colored the questions, and Americans reacted differently based on whether they were given the impression that the United States was acting unilaterally or not. This is an enormous source of systematic error—an infinitesimal shift in phrasing can completely change how people answer a poll question. Even something as seemingly insignificant as the order of questions can make a tremendous difference.
When there’s a tricky issue likely to cause problems, some of the better-funded polls ask two or more sets of people one of two or more different versions of the same question to try to ensure that the wording isn’t messing with the results.44 Unfortunately, this makes the poll more expensive and increases the margin of error, since splitting a question into two different variants means that each variant gets answered by half the number of subjects. (Worst of all from the perspective of the polling organization, if two phrasings get vastly different answers, it might reveal that the poll is meaningless.) Thus most polls don’t control for systematic errors caused by wording, even when it’s desperately needed.
And it is desperately needed quite often. Anytime there’s a politically sensitive or awkward issue, the pollster’s choice of language is