Academic Legal Writing - Eugene Volokh [97]
But here's another fact, noted in the same item several paragraphs later: “Today, according to the union's own optimistic estimates, only about 30 percent of all mines are organized.” This means nonunion mines make up at least about 70 percent of all mines. Even if nonunion mines were just as safe as union mines, it would make sense that 70 percent of all mines would account for “[t]he majority of mining deaths.” The article didn't just fail to show that lack of unionization causes a decrease in safety; it even failed to show that lack of unionization is correlated with a decrease in safety.
Now it well may be that both correlation and causation are present. For instance, perhaps the 30% of all union-organized mines contain 80% of all miners, and yet account only for 25% of all mining deaths. Perhaps the unionization is the reason for the greater safety.
The trouble is that the magazine article did not show this. It gave only two pieces of data: Over 50% of mining deaths were in nonunion mines, and 70% of mines were nonunion mines. That doesn't support the author's conclusion that “The real obstacle to safety reform is that miners no longer have a powerful union sticking up for them.”
5. Beware of “10% of all Xs are responsible for 25% of all Ys” comparisons
People commonly report that a few supposed bad actors are responsible for a disproportionate share of bad behavior. One unpublished paper I read, for instance, reported that 10% of all police officers in a department accounted for 25% of all abuse complaints, and used that as evidence that some police officers are especially prone to misbehavior.
But this data point, standing alone, is entirely consistent with all police officers being equally prone to behave badly. Even if all officers are equally likely to misbehave, and complaints are randomly distributed across all officers, we could easily see the 10%/25% distribution, or many other lopsided distributions.
Consider a boundary case: Say that each police officer has a 10% chance of being the subject of a complaint this year. Then, on average, 10% of all officers will yield 100% of this year's complaints. Yet by hypothesis all the officers are equally prone to having complaints filed against them.
Likewise, say that there is a 1% chance of a complaint against each police officer each year for 10 years, and the probabilities are independent from year to year (because complaints are entirely random, and all the officers are equally prone to them). Then, on average 9.5% (1-0.9910) of all police officers will have 100% of the complaints over the 10 years, since 90.5% (0.9910) of the officers will have no complaints.
Or say that each police officer has a 10% chance of having a complaint each year, and we're again looking at results over 10 years. Then 7% of all officers will have 3 or more complaints—but those 7% will account for 22.5% of all complaints.* Again, this is so even though each officer is equally likely to get a complaint in any year.
We can likewise see this statistical phenomenon in situations where there's no doubt that the distribution is random. Say, for instance, that you have 100 coins, each of which is 50% likely to turn up heads and tails. You toss each coin twice. On average,
a. 25 of the coins will come up heads twice, accounting for 50 heads.
b. 50 of the coins will come up heads once and tails once, accounting for 50 heads.
c. 25 of the coins will come up tails twice, accounting for no heads.
This means that 25% of the coins (the ones in the first group) account for 50% of the heads—but because of randomness, not because some coins are more likely to turn up heads than others. The same would happen if you tossed 10,000 coins; this is not a problem that can be avoided by increasing the sample size.
Now of course some officers will indeed be more prone to complaints than others. And other kinds of