unSpun_ Finding Facts in a World of Disinformation - Brooks Jackson [52]
APPEALS TO AUTHORITY
Linus Pauling won two Nobel Prizes, one for chemistry and the other for peace, but he had no particular expertise in medicine. Nevertheless, millions of people swallowed his claims that high doses of vitamin C could cut the incidence of the common cold and might even be effective against cancer. In fact, at least sixteen controlled experiments, some involving thousands of volunteers, have failed to show that vitamin C has any effect on either condition. Dr. Stephen Barrett of Quackwatch.org reports that “no responsible medical or nutrition scientists share [Pauling’s] views.”
* * *
FactCheck.org’s Guide to Testing Evidence
At FactCheck.org, we find ourselves asking a few basic questions again and again in evaluating evidence. Here’s a short list of tests we have found useful:
• Is the source highly regarded and widely accepted? There are a number of longstanding organizations we know we can count on for reliable, unbiased information. For job statistics, the Bureau of Labor Statistics is every economist’s basic source. For hurricane statistics, the National Hurricane Center is considered the authority. For determining when business recessions begin and end, the private, nonprofit National Bureau of Economic Research is an authority cited in economic history books. And for abortion statistics, the Guttmacher Institute is accepted by all sides as trustworthy.
• Is the source an advocate? Claims made by political parties, candidates, lobbying groups, salesmen, and other advocates may be true but are usually self-serving and as a result may be misleading; they require special scrutiny. Always compare their information with other sources. The National Research Council—chartered by Congress and able to call on the nation’s most eminent experts—is a better source on whether gun-control laws cut crime than is a group devoted to lobbying for or against such laws. The Guttmacher Institute does advocate “reproductive freedom,” but we accept its numbers not only because the institute is trusted by both sides of the debate, but also because in this case they support a president who is on the other side of the issue. The institute’s report that the number of abortions continues to diminish under Bush is the opposite of what we would expect if it were allowing ideological bias to color its findings.
• What is the source’s track record? Look for previous experience. In our abortion example, Stassen had no record of conducting studies on abortion statistics. In contrast, the Guttmacher Institute’s surveys of abortion providers go back to 1973.
• What method is used? Mitch Snyder’s estimate of the number of homeless people turned out to depend on a collection of guesses, at best. The Urban Institute’s estimates, though they also involve some assumptions and guesswork, are based on U.S. Census data gathered in a uniform way from a very large, random sample.
• Does the source “show its work”? Good researchers always explain how they arrived at their numbers and conclusions. Daniel Cristol described exactly how he and his colleagues conducted their 400 timed observations of crows, and he published the results. Good research methods are transparent.
• Is the sample random? News organizations and websites are fond of conducting “unscientific” polls. Viewers or visitors are asked to express a preference, and the results are reported. This is just a marketing method designed to draw interest; the results are utterly meaningless because the sample is self-selected, not random. Some such polls have even been intentionally rigged. At best, they are like Stassen’s nonrandom sixteen-state sample, which turned out not to reflect the situation in all fifty states.
• Is there a control group? Good scientific procedure requires a “control” to provide a valid basis for comparison. A crow