Proofiness - Charles Seife [73]
Every poll relies upon the cooperation of its subjects—a poll can’t record the opinions of people who toss their reply card in the trash or who slam down the phone when they hear the voice of a pollster. As a result, all polls are subject to “volunteer bias” that can inject an enormous amount of error into the poll. The census is no different. Every single household in the United States gets a census form, and the majority fill it out, but quite a few don’t. In 2000, roughly one in three households didn’t bother to return their questionnaire. To get an accurate count of the population, the Census Bureau still has to count the citizens in the households that refused to respond. This is where the big spending comes in. The bureau dispatches thousands of census workers who spend months going from household to household tracking down nonrespondents. The harder it is to get a household to respond, the more money is spent to try to contact the people in that household, but the bureau keeps trying until they run out of time and are required, by law, to give Congress the results. By the end of the process, the bureau manages to wring data out of all but about 2 percent of the population. It’s an extraordinary effort. But it still is full of errors.
Not only has the census failed to reach 2 percent of the population; it accidentally double-counts about 1 percent. This means that for all that effort, the census is only good to within about ten million people, plus or minus. This plus or minus is enormously important, politically; these ten million people would be entitled to roughly fourteen representatives in the House. It’s incredibly disheartening; all that time and money spent, and errors in the census are still huge. These errors are impossible to correct by ordinary means. The government could theoretically stake out the homes of every single nonrespondent, but that would cost astronomical amounts of money, and even this wouldn’t manage to catch everybody. Even with double its current budget, the Census Bureau can’t do much better with its measurements than they already are. However, the situation isn’t hopeless. There is a way to reduce these errors enormously by using a set of statistical tricks known collectively as sampling.
The best way to understand sampling is through an example. Imagine that there’s a shallow pond that’s full of trout and minnows. The government has hired you to count how many fish the pond contains. You row gently from one end of the pond to the other, counting the fish that you see along the way. You come up with a count of 599 trout and 301 minnows. Your grand total is 900 fish in the pond, about 67 percent of which are trout and 33 percent minnows.
As you can probably guess, the answer is off because your count is error-prone. One source of error is that the fish are constantly moving about, making it all but certain that you’ll count some fish twice and others not at all. Another source of error is that minnows are harder to spot than trout. They’re tiny and timid; they tend to hide when the boat comes nearby. So it’s quite likely that you’re undercounting minnows—and no matter how many times you count from your boat, minnows are likely to be underrepresented. Conversely, big, visible trout are more likely to be double-counted.
You can correct for these errors, but to do so, you have to make another measurement to figure out how bad they really are. After you’ve done your initial survey of aquatic life, you do another boat count of a small, representative section of the pond and record the numbers of fish that you find (say, 30 trout and 15 minnows). Then you make a more careful (and more invasive) count of that small section. Net off that little region of the pond, dredge up every single fish in that area, and pull them into the boat. Counting them as you toss them one by one back into the pond yields an incredibly