The 4-Hour Body_ An Uncommon Guide to Ra - Timothy Ferriss [204]
In other words, you better make sure that 20% holds up with at least 453 flips with each coin. In this case, 10 extra flips out of 100 doesn’t prove cause-and- effect at all.
Three points to remember about p-values and “statistical significance”:
• Just because something seems miraculous doesn’t mean it is. People are fooled by randomness all the time, as in the birthday example.
• The larger the difference between groups, the smaller the groups can be. Critics of small trials or self-experimentation often miss this. If something appears to produce a 300% change, you don’t need that many people to show significance, assuming you’re controlling variables.
• It is not kosher to combine p-values from multiple experiments to make something more or less believable. That’s another trick of bad scientists and mistake of uninformed journalists.
TOOLS AND TRICKS
The Black Swan by Nassim Taleb (www.fourhourbody.com/blackswan) Taleb, also author of the bestseller Fooled by Randomness, is the reigning king when it comes to explaining how we fool ourselves and how we can limit the damage. Our instinct to underestimate the occurrence of some events, while overestimating others, is a principal cause of enormous pain. This book should be required reading.
The Corporation, DVD (www.fourhourbody.com/corporation) This is a disturbing documentary about the American corporation and its relentless pursuit of profit at the expense of our culture. This film gives you a glimpse into how heavily companies can skew health reports when they have a vested interest in the findings. See the next chapter.
“List of Cognitive Biases” (www.fourhourbody.com/biases) We are all susceptible to cognitive biases, including the scientists who produce “bad science.” Review the list at this URL and ask yourself whether you’re mindlessly accepting as fact things you hear or read.
End of Chapter Notes
3. Also called population, cohort, or epidemiological studies.
4. The one exception is if the effect is so huge that it can’t be explained in any other way. For instance, the twentyfold increased risk of lung cancer that is associated with cigarette smoking in multiple studies.
5. Michael Pollan, “Unhappy Meals,” New York Times, January 28, 2007, sec. Magazine.
6. Caloric-value-of-100g/100g = X/number-of-grams-weighed.
7. Even without such tools, if you have large samples and the analysis is good, it is sometimes possible to correct for subjective error and reconstruct the information you want.
SPOTTING BAD SCIENCE 102
So You Have a Pill …
This chapter was written by Dr. Ben Goldacre, who has written the weekly “Bad Science” column in the Guardian since 2003 and is a recipient of the Royal Statistical Society’s Award for Statistical Excellence in Journalism. He is a medical doctor who, among other things, specializes in unpacking sketchy scientific claims made by scaremongering journalists, questionable government reports, evil pharmaceutical corporations, PR companies, and quacks.
What I’m about to tell you is what I teach medical students and doctors—here and there—in a lecture I rather childishly call ‘Drug Company Bullshit’. It is, in turn, what I was taught at medical school,1 and I think the easiest way to understand the issue is to put yourself in the shoes of a big pharma researcher.
You have a pill. It’s OK, maybe not that brilliant, but a lot of money is riding on it. You need a positive result, but your audience aren’t homeopaths, journalists or the public: they are doctors and academics, so they have been trained in spotting the obvious tricks, like ‘no blinding’, or ‘inadequate randomisation’. Your sleights of hand will have to be much more elegant, much more subtle, but every bit as powerful.
What can you do?
Well, firstly, you could study it in winners. Different people respond differently to drugs: old people on lots of medications are often no-hopers, whereas younger people with just one problem are more likely to show an improvement. So only study your drug in the latter group. This will make your research