The 4-Hour Body_ An Uncommon Guide to Ra - Timothy Ferriss [207]
End of Chapter Notes
1. In this subject, like many medics of my generation, I am indebted to the classic textbook How to Read a Paper by Professor Greenhalgh at UCL. It should be a best-seller. Testing Treatments by Imogen Evans, Hazel Thornton, and Iain Chalmers is also a work of great genius, appropriate for a lay audience, and, amazingly, also free to download from www.jameslindlibrary.org. For committed readers I recommend Methodological Errors in Medical Research by Bjorn Andersen. It’s extremely long. The subtitle is An Incomplete Catalogue.
THE SLOW-CARB DIET—194 PEOPLE
The following Slow-Carb Diet data was collected with detailed questionnaires using CureTogether.com. 194 people responded to all questions, and 58% indicated it was the first diet they had ever been able to stick with.
The subjects were recruited via my top-1,000 blog (www.fourhourblog.com), Twitter (www.twitter.com/tferriss), and Facebook (www.facebook.com/timferriss).
Average Weight Lost (lbs.) Number of People
Everyone 21 194
Vegetarian 23 10
Nonvegetarian
21
178
Age
15–20 16 19
21–30 20 86
31–40 22 56
41–50 21 26
51–60 30 5
61+ 11 2
Men
23
150
Women
12
44
Kids
21
60
No kids 20 118
First-week loss
3.4
194
Second-week loss 3.1 194
Third-week loss 3.3 194
Fourth-week loss 4.0 194
Skipped breakfast
23
29
Did not skip breakfast 21 157
Had breakfast within one hour 20 127
Did not have breakfast within one hour 23 61
Meals per day
Two 39 8
Three 19 80
Four 20 64
Five 23 36
Followed the diet strictly
18
84
Modified the diet 23 104
Counted calories
27
35
Did not count calories 20 152
Exercised while dieting
22
144
Did not exercise while dieting 18 41
Started exercise after starting diet
25
68
Did not start exercise after starting diet 19 116
Women with children
12
16
DISTRIBUTION OF RESULTS
NUMBER OF PEOPLE
Gained 0–22 lbs 4
Lost 0–10 lbs 39
Lost 11–20 lbs 68
Lost 21–30 lbs 35
Lost 31–40 lbs 16
Lost 41–50 lbs 11
Lost >50 lbs 10
(Potential) Weaknesses of the Data
The data here, while fascinating, are not perfect. Here are two stand-out weaknesses of the methodology, and of polls in general:
PEOPLE COULD BE MAKING THINGS UP.
Though we removed obvious duplicates, omitted garbage data (“I lost 650 pounds!,” “I weighed 35 pounds at the beginning,” etc.), and flagged questionable entries, no one was checking IDs or making in-person visits. Unless you’re conducting a controlled trial, it’s hard to avoid this problem.
THIS DATA SET MIGHT NOT ACCOUNT FOR DROPOUTS—THE PEOPLE WHO TRIED AND GAVE UP.
Given the number of failures in the 3,000 comments reviewed, between 3% and 5%, one would expect more failures reported. Of 194 respondents, only four gained weight or remained the same. Don’t forget: these respondents were reached after leaving comments on a blog post or self-selecting by responding to Twitter or Facebook.
The challenge of the missing dropouts belies a common weakness with questionnaires that are open to the public: those most likely to respond are often those who have had positive results.8 This is a form of survivorship bias, a concept well worth understanding.
Looking at average mutual fund returns from last year to pick a winner? Don’t forget that you are asking the survivors. The casualties—what Nassim Taleb refers to as “silent evidence”—aren’t around to be polled. The “average” returns are less impressive if you can include the people who bet the farm and lost. Finding those dead bodies is hard, especially in finances, when there is so much incentive to cover them up.
In practical terms, does this mean our diet results are bogus? Not at all. The possibility of survivorship bias isn’t proof that the numbers aren’t representative. Two things to keep in mind: