Proofiness - Charles Seife [38]
That faith is symbolized by a little number that accompanies every major poll. Journalists interpret this number as a conditional guarantee of a poll’s truth—it encapsulates all the uncertainty about a poll’s veracity and rolls it into a tiny little ball that can be ignored. This number is arguably the most misunderstood and abused mathematical concept that journalists have gotten their fingers on: the margin of error.
Ask a bunch of journalists what a margin of error is, and you’ll get a whole lot of contradictory answers. However, reporters tend to treat the margin of error as a litmus test about whether or not to believe the results of a poll.
For example, if a poll shows that 51 percent of the public would vote for George W. Bush and 49 percent would vote for John Kerry, it looks like a very slim two-point lead for Bush. No doubt many pundits would describe the result that way. However, the more cautious journalists in the newsroom wouldn’t accept that result blindly. They would compare the difference between the two candidates to the poll’s margin of error, which, for reasons to be explained shortly, typically hovers around 3 percent. Once they discover that the lead—2 percent—is less than the margin of error—3 percent—they typically mumble an incantation about statistical significance and reject the lead, declaring the poll to be a dead heat. If, on the other hand, the difference between the two candidates is greater than the margin of error, then the lead is real: the poll’s result is to be believed. Journalists use the margin of error as a touchstone, as a nearly infallible guide that allows them to separate good polls from bad; valid data from dross. Journalists’ faith in this test is somewhat surprising, given how few of them understand what a margin of error really represents.
The margin of error is a rather subtle concept that describes a kind of problem that plagues polls. At its core, a poll is a measuring instrument designed to gauge public opinion, and, like any other measuring instrument, a poll is imperfect. Polls are inherently error-prone. However, there’s a peculiar type of error that stems from the way that a poll is conducted—one that springs from randomness.
To make a measurement with a poll, a pollster asks a set of people—the “sample”—some questions. From the answers, the pollster knows, at least in theory, what the sample believes about a certain burning issue. With that data, the pollster then makes a leap of faith. The pollster assumes that the preferences of that small sample of people accurately represent the predilections of the entire population—the “universe.” That is, if the pollster queries a sample of, say, a thousand people in Britain and finds that 64 percent prefer tea to coffee, she concludes that roughly the same thing is true of the entire universe of British citizens: she reports to an eager public that roughly 64 percent of Britons prefer tea to coffee. Unfortunately, this leap of faith introduces an unavoidable error. Even under the very best of circumstances, even when the pollster is extremely careful about selecting a sample of people to poll, the randomness of nature conspires to prevent her from getting a precise answer.
This error makes sense if you think about it; there has to be some kind of error associated with the size of the sample in a poll: the smaller the sample, the bigger the error and the less reliable it will be. A sample size of one is all but useless—asking a single random stranger what he likes to drink is not going to give you a fair cross section of what all of Britain enjoys. You might have randomly stumbled across a loon who prefers drinking