Academic Legal Writing - Eugene Volokh [86]
“The right to privacy” is a notoriously ill-defined phrase, but the same can happen with other questions. For instance, another question asked people:
Many college and university professors currently have the academic freedom to take controversial stands in their classrooms and to publish controversial materials in books and journals. Would you favor or oppose restrictions on the academic freedom of professors to criticize government military policy during times of war?
41% favored such restrictions, and 56% opposed them.
Unfortunately, there are two questions combined into one here: whether what professors say in class should be restricted and whether what professors publish outside class should be restricted. (The courts certainly recognize them as different questions—the First Amendment right of public university professors to speak outside class is wellestablished, but there's a hot debate about whether they have similar rights inside class, especially when their speech is only tangentially related to the subject matter of the class.49) Some respondents may have understood the question as focusing mainly on in-class speech. Others may have understood it as focusing mainly on books and articles. Others may have thought it focused on both, and took the same view as to both. Others may have thought it focused on both, and had no way of expressing their different views about both.
So it is a mistake to report, as some newspapers did, that “41 percent [of respondents] said university professors should be restricted from criticizing U.S. military policy during wartime.” We don't know what fraction of the respondents actually thought professors should be restricted from criticizing U.S. military policy altogether, and what fraction thought only that professors should be restricted from using their classrooms to do so—a distinction that judges find significant, and that many readers might, too.
c. Get the text of the questionnaire
To avoid these problems, you should get the text of the questions used in any survey on which you want to rely. Many survey organizations release their questionnaires on the Web together with the results (though many media outlets don't fully quote this text); others will give you the questions if you ask them nicely. You should in turn include the text of the relevant question in either the body of the article or the footnotes, where you cite the survey.
If a survey organization refuses to release the questions, then you should be skeptical of the survey's accuracy. You probably shouldn't cite such a survey, and if you do, you should at least alert the reader that the organization refused to release the questions, and that the results of the survey are thus especially hard to evaluate.
4. Errors caused by ignoring information from the same survey
Surveys, especially sophisticated ones, often ask many questions and yield a great deal of information. If you use such a survey, it's your responsibility to make sure that you consider all this information, and not just the parts that seem to support your case. And if you rely on a secondary source that uses such a survey, it's your responsibility to make sure that the source has used the survey properly.
Let me return to the State of the First Amendment survey, and an article that summarizes it as follows:
Many Americans, spooked by the Sept. 11, 2001, terrorist attacks on their country, seem inclined to clamp down on First Amendment freedoms, especially freedom of the press ....
Each year, the First Amendment Center in Nashville, Tenn.—an independent affiliate of The Freedom Forum—conducts a survey of Americans' attitudes toward the First Amendment ....
Among the findings:
* About 49 percent said the First Amendment gives us too much freedom, up from 39 percent last year and