Academic Legal Writing - Eugene Volokh [181]
2. The Sourcebook of Criminal Justice Statistics Online table (see below) does say that 69.4% of all sexual abuse offenses in its dataset were committed by “Native Americans, Alaska Natives, Asians, and Pacific Islanders.” But the table reports only federal prosecutions, as the heading “Offenders sentenced in U.S. District Courts” and the fourth line of the Note reveal.
Nearly all sexual abuse cases are prosecuted in state court; the main federal law covering sexual abuse is the one that applies to Indian reservations. American Indians thus commit a tiny fraction of all sexual abuse nationwide, but they commit a large fraction of the sexual abuse prosecuted by the federal government. Understand what jurisdiction your data covers.
C. USA Today Survey Report, p. 170
As the problem mentioned, the question in this graphic refers to a Ninth Circuit case that concluded that the use of the words “under God” in the Pledge of Allegiance violates the Establishment Clause.
1. The first problem isn't with the statistics: The court of appeals didn't rule that “the Pledge ... is unconstitutional”; it ruled that the inclusion of “under God” in the Pledge is unconstitutional. This means that though the current text of the Pledge is impermissible, the Pledge could still be said with two words out of about 30 excised. Simply calling the decision a “ruling that the Pledge of Allegiance is unconstitutional” is likely to mislead many readers.
2. From the text of the headline (“Most say ‘Pledge’ is constitutional”), most of whom did you think said this? When a national paper says “most say,” most readers will assume “most Americans,” “most citizens,” or some such. But the tiny type in the bottom of the box says, “Source: JD Jungle online survey of 235 law students and legal associates June 26–27. Margin of error: ±3 percentage points.”
So the survey only measured the views of law students and legal associates (whatever “legal associates” exactly means), not a representative sample of the public. Is “Most say ‘Pledge’ is constitutional” / “Yes 27% / No 73%”—in a paper that isn't aimed at lawyers—an accurate way of summarizing a poll of an uncertain subset of the legal profession?
3. Beyond this, the poll isn't even a valid estimate of the views of “law students and legal associates.” The poll is an “online survey,” so it's not a random sample, but a self-selected one: It registers only the votes of those people who hear about the survey and care about it enough to participate—likely those who are unusually interested in the subject, and not a representative sample of any group.*
4. Finally, even a random sample of 235 people couldn't yield a margin of error of ±3% (an assurance that there's a 95% chance that the reported result is within 3% of the true breakdown of people's views). If you divide 100 by the square root of 235, you get a margin of error of roughly ±6.5%, and if you're more precise and follow the instructions in the footnote on p. 162, you'll get roughly ±5.8%.
The margin of error only makes sense for randomly chosen samples, not for self-selected ones—but even if we ignore that problem, the ±3% margin of error is incorrect.
D. Drunk Driving Study, p. 182
Recall the exercise: Assume that a study showed that 15% of New York drivers aged 16 to 25 drive drunk at least once a month. The Minnesota legislature is considering new penalties for drunk driving by 16–to–18–year-olds, and a commentator who supports the law writes “Drunk driving has reached epidemic proportion among teenagers, with 15% of driving-age teenagers driving drunk at least once a month.” What errors or unstated assumptions can you find in this statement?
1. Extrapolating from one place and time to another: