Academic Legal Writing - Eugene Volokh [84]
d. Bad samples: self-selected samples
The media often publish so-called “self-selected” surveys: For instance, USA Weekend once ran a reader poll which asked whether readers thought the nation would be safer if all law-abiding adults were entitled to get a license to carry a concealed weapon. It got 34,000 responses, of which 82% said “yes,” a stunning majority in favor of gun decontrol.47
But this number is meaningless. First, this sample is obviously biased in one way: It measures only the views of USA Weekend readers. Beyond this, though, the survey doesn't even tell us what the average USA Weekend reader thought, because only a small and likely unrepresentative fraction of those readers responded. Who takes the time, effort, or money to answer one of these surveys? Likely the people who feel most strongly about the survey topic, and not just average newspaper readers. What's more, many activists tend to e-mail their friends about these polls, so groups that are particularly well-organized on the Internet can quickly swamp the poll results.
Note, incidentally, that the large size of the sample was irrelevant. If you get 34,000 self-selected responses, the result tells you nothing about the views of the larger group. But, if you select even 1000 people randomly from the country at large, you can get results that are accurate within a range of ±3% (if you do other things right).
e. Bad samples: mail-in samples or Internet samples
Most mail-in polls and virtually all Internet polls involve self-selected samples, since so few people tend to respond to them. This was another problem with the Literary Digest poll: Only 25% of the people who were sent surveys responded, and the respondents' views ended up being quite unrepresentative even of all those who got the surveys.48
A mail survey might be made valid, if the survey-takers follow up with all the people who didn't respond, and ultimately get a fairly high response rate. But it's virtually impossible to make a Web-based survey be valid.
f. What makes a sample sound
So which surveys are indeed valid? First, the survey-takers must try to reach a random sample of a broader group. Second, they must get responses from a majority of the people whom they're trying to reach, to avoid self-selection bias; the best surveys usually have response rates of 70% or above.* Third, they must have a large enough number of respondents to yield a fairly small margin of error: Remember that you'd need 1000 respondents for a ±3% margin, and having a mere 100 respondents will yield a ±10% margin, which is rarely accurate enough.
Most useful surveys involve either random-digit dialing of phone numbers or exit polls (though even exit polls have had serious problems). As I mentioned, the phrases “online survey” and “Internet poll” are almost sure signs of invalidity.
Note that it's especially hard to do surveys of relatively small subgroups of the population, such as Jews or Asians. There's no master list of Jews from which one can draw a random sample, so the best way to poll Jews is to choose a random sample of the population at large, ask respondents whether they are Jewish, and record the answers of the Jews separately from those of the non-Jews. But Jews only make up about 2% of the population, so to get a sample of even 400 people (enough for a ±5% margin of error) you'd need to call 20,000 people—an expensive undertaking.
Some sophisticated polling techniques might make the task more manageable, but it still wouldn't be easy; and, in addition, some respondents might not want to reveal their religion or ethnicity to a stranger. So most polls that purport to measure the views of small subgroups tend to have very high margins of error for those