Academic Legal Writing - Eugene Volokh [83]
relying on National Safety Council's Injury Facts (2000) (1996 data), p. 17, which I reproduce below. (The actual article relied on an intermediate source.) What's wrong with the quote? The answer is on p. 361.
2. The table on the next page, from the Sourcebook of Criminal Justice Statistics, seems to show that 69.4% of all sexual abuse offenses are committed by “Native Americans, Alaska Natives, Asians, and Pacific Islanders,” who together make up 5% of the population.45 What's the explanation? The answer is on p. 361.
G. Handle Survey Evidence Correctly
1. What do surveys measure?
Survey evidence is often indispensable, and can be fairly reliable. But many surveys are conducted badly, and even well-conducted surveys are often misinterpreted as measuring things that they don't in fact measure. To avoid relying on bad surveys and misrepresenting good ones, we need to ask: What exactly do surveys measure?
Most precisely, surveys measure only what (1) the survey-takers recorded (2) these particular respondents (3) were willing to say (4) in response to the particular questions they were asked—not very useful. But it also turns out that surveys of a small group can reveal to us the likely answers of a larger group, if (and only if) the respondents are a large enough randomly selected sample of the broader group; and this can be far more useful, if it's done right. Understanding these limitations of surveys should help us identify several ways that one can err in using surveys, and thus help us figure out how to avoid such errors.
2. Errors in generalizing from the respondents to a broader group
a. Why proper sampling can yield generalizable survey results
Surveying a randomly selected sample of a group gives us results that are pretty generalizable to the whole group. And the survey's accuracy is closely related to the absolute number of people who are asked, not to what percentage of the broader group is surveyed.
That's why you can get a good sense of the views of 280 million Americans by asking even as few as 1000 people. With a randomly selected group of 1000 people, and results ranging from 50%–50% to 80%–20%, you generally get a “margin of error” of ±3%, which means that there's a 95% chance that the actual views of the population at large are within ±3% above or below the result you get from the survey: If the survey says that 42% of respondents say they believe something, there's a 95% chance that the actual number of people who would say they believed that is between 39% and 45%.* The margin of error ends up being roughly 100% divided by the square root of the sample size, so at 100 people the margin is ±10%, and at 2500, it's ±2%.
But—and here's the single most important thing you should remember about surveys—this only works if the respondents are a randomly selected sample of the whole group. If the respondents are not a randomly selected sample, or very close to it, then it is mathematically impossible to draw an inference from their responses to the likely responses of the whole group.
Unfortunately, several common sample selection techniques violate this assumption.
b. Bad samples: Biased samples
One of the great cautionary tales of survey-taking comes from the 1936 presidential election. The election was won in a 61%–37% landslide by Franklin Roosevelt over Alf Landon, but a vast (2-millionperson) Literary Digest poll conducted in the weeks before the election showed Landon getting 55% of the vote and Roosevelt 41%.
Part of the problem was simple: The Literary Digest pollsters found people's addresses primarily from telephone books and automobile registration records—which means they disproportionately polled richer people. The views of these richer voters may have been quite unrepresentative of the views of all voters.46
c. Bad samples: convenience samples
A special case of the biased sample problem is the so-called “convenience sample”—a group of people chosen because they're convenient, such as a professor's freshman psychology students, or a group of pedestrians who