Academic Legal Writing - Eugene Volokh [85]
3. Errors in generalizing from the question being asked
a. Surveys that ask a different question
A survey can at best measure people's views on the particular question that was asked; to accurately use a survey, you must therefore properly identify that question. Consider, for instance, how several newspaper and magazine articles summarized the First Amendment Center's State of the First Amendment 2002 report; I quote one in particular:
The First Amendment goes too far in guaranteeing free speech, say 49 percent of people polled by the First Amendment Center. The percentage of people who think speech protections are too robust is up some 10 points from 2001.
Seems pretty striking, no? But here's the question that the survey asked:
The First Amendment became part of the U.S. Constitution more than 200 years ago. This is what it says: “Congress shall make no law respecting an establishment of religion or prohibiting the free exercise thereof; or abridging the freedom of speech, or of the press; or the right of the people peaceably to assemble, and to petition the government for a redress of grievances.” Based on your own feelings about the First Amendment, please tell me whether you agree or disagree with the following statement: The First Amendment goes too far in the rights it guarantees.
This question thus did not measure people's attitudes towards “guaranteeing free speech,” but rather people's attitudes towards the First Amendment as a whole, including the Religion Clauses. It's impossible to tell from this question how many of the 49% thought the speech protections went too far, and how many only thought this about one of the other protections.
And there's good reason to think that much less than 49% of respondents really thought that “speech protections are too robust.” A later question in the survey asked people “Overall, do you think Americans have too much freedom to speak freely, too little freedom to speak freely, or is the amount of freedom to speak freely about right?”; only 10% said “too much” (67% said “about right,” 21% said “too little,” and 1% said they didn't know, or refused to answer). Moreover, some of the respondents were questioned in the days after a Ninth Circuit panel interpreted the Establishment Clause as forbidding schools from using the words “under God” in the Pledge of Allegiance, so it's likely that many respondents were more focused on the establishment of religion than on the freedom of speech. But even setting these speculations aside, saying that 49% of respondents believe that “[t]he First Amendment goes too far in guaranteeing free speech” is just a misreading of what the survey actually asked.
b. Surveys that ask ambiguous questions
If a survey asks questions that different people are likely to interpret differently, then it can't really measure anything in particular.
The State of the First Amendment survey, for instance, also asked the following:
The U.S. Constitution protects certain rights, but not everyone considers each right important. I am going to read you some rights guaranteed by the U.S. Constitution. For each, please tell me how important it is that you have that right .... [H]ow important is it that you have the right to privacy?
81% of the respondents said this was essential, and 18% more said it was important; only 1% said it was not important.
Unfortunately, this tells us little about people's actual views. The hottest debates about the right to privacy are, of course, about abortion; but obviously many respondents did not interpret the right to privacy as covering abortion rights, since far more than 1% of the public believes that the Constitution should not be read as protecting abortion.
Some people must have thought that “the right to privacy” refers to something else—perhaps the right to be free from unreasonable searches and seizures, or the many other things that are sometimes referred to as “the right to privacy” (some of which are limitations on the