The 4-Hour Body_ An Uncommon Guide to Ra - Timothy Ferriss [201]
Observational studies are valuable for developing hypotheses (educated guesses that can then be tested in controlled settings) but they cannot and should not be used to show cause and effect. To do so is both irresponsible and potentially dangerous.
3. DOES THIS STUDY DEPEND ON SELF-REPORTING OR SURVEYS?
In 1980, scientists at an isolated research station in Antarctica had test subjects weigh and record all of the food they consumed. Once a week, the subjects were asked to recall what they had eaten the day before (food, keep in mind, they had weighed and recorded in their notebooks). Despite all the attention to logging meals, the men still underestimated their intake by 20–30%.
Granted, these might have been blizzard-blind men. Hardly normal circumstances. Let’s look at how the real professionals do it.
The Women’s Health Initiative (WHI) was a massive $415-million project conducted under the auspices of the National Institutes of Health (NIH) and involving nearly 49,000 women. It was designed to investigate a number of health issues, including the effect of low-fat diets on cancer, over an eight-year period. It was a large-scale intervention study, often viewed by media as the gold standard in nutrition research. Dr. Michael Thun of the American Cancer Association went so far as to call the WHI “the Rolls-Royce of studies.”
The New York Times announced the results in 2006 with a headline that was crystal clear:
“LOW-FAT DIET DOES NOT CUT HEALTH RISKS, STUDY FINDS”
Though I agree with the conclusion based on other data, the WHI study couldn’t possibly conclude this. Let’s look to Michael Pollan for a peek under the hood at one of the most common weaknesses of nutritional studies: self- reporting.
To try to fill out the food-frequency questionnaire used by the Women’s Health Initiative, as I recently did, is to realize just how shaky the data on which such trials rely really are.… It asked me to think back over the past three months to recall whether when I ate okra, squash or yams, were they fried, and if so, were they fried in stick margarine, tub margarine, butter, “shortening” (in which category they inexplicably lump together hydrogenated vegetable oil and lard), olive or canola oil or nonstick spray? I honestly didn’t remember, and in the case of any okra eaten in a restaurant, even a hypnotist could not get out of me what sort of fat it was fried in.…
This is the sort of data on which the largest questions of diet and health are being decided in America today.
Other WHI questions include:
When you ate chicken or turkey, how often did you eat the skin?
Did you usually choose light meat, dark meat, or both?
In the last three months, how many times have you eaten a half-cup serving of broccoli?5
How would you do? Think you can pull off recollection within 20% of reality?
Let’s test the last 24 hours.
Do this: estimate the number of calories you ate and drank yesterday, without using any references, as well as the fat calories you consumed. Then eat the same meals and snacks, but weigh it all in grams on a portable kitchen scale. Measure drinks using a measuring cup. Use www.nutritiondata.com to determine the 100-gram caloric value of each food and do the math.6
Unless you are a professional athlete and can distinguish between a 150-gram serving of steak and a 200-gram serving, the difference between your self-reporting and your weighing analysis will shock you. Combine that misreporting across 49,000 people and you can imagine the rather Picasso-like picture it creates.
Granted, self-reported data can be useful, especially when recording in real time (i.e., recording something as it happens). This has become easier with technology that takes memory out of the equation, such as DailyBurn’s FoodScanner iPhone