Knocking on Heaven's Door - Lisa Randall [103]
Uncertainty doesn’t mean that scientists treat all options or statements equally (though news reports frequently make this mistake). Only rarely are probabilities 50 percent. But they do mean that scientists (or anyone aiming for complete accuracy) will make statements that tell what has been measured and what it implies in a probabilistic way, even when those probabilities are very high.
When scientists and wordsmiths are extremely careful, they use the words precision and accuracy differently. An apparatus is precise if, when you repeat a measurement of a single quantity, the values you record won’t differ from each other very much. Precision is a measure of the degree of variability. If the result of repeating a measurement doesn’t vary a lot, the measurements are precise. Because more precisely measured values span a smaller range, the average value will more rapidly converge if you make repeated measurements.
Accuracy, on the other hand, tells you how close your average measurement is to the correct result. In other words, it tells whether there is bias in a measuring apparatus. Technically speaking, an intrinsic error in your measuring apparatus doesn’t reduce its precision—you would make the same mistake every time—though it would certainly reduce your accuracy. Systematic uncertainty refers to the unbeatable lack of accuracy that is intrinsic to the measuring devices themselves.
Nonetheless, in many situations, even if you could construct a perfect measuring instrument, you would still need to make many measurements to get a correct result. That is because the other source of uncertainty50 is statistical, which means that measurements usually need to be repeated many times before you can trust the result. Even an accurate apparatus won’t necessarily give the right value for any particular measurement. But the average will converge to the right answer. Systematic uncertainties control the accuracy of a measurement while statistical uncertainty affects its precision. Good scientific studies take both into account, and measurements are done as carefully as possible on as large a sample as is feasible. Ideally, you want your measurements to be both accurate and precise so that the expected absolute error is small and you trust the values you find. This means you want them to be within as narrow a range as possible (precision) and you want them to converge to the correct number (accuracy).
One familiar (and important) example where we can consider these notions is tests of drug efficacy. Doctors often won’t say or perhaps they don’t know the relevant statistics. Have you ever been frustrated by being told, “Sometimes this medicine works; sometimes it doesn’t”? Quite a bit of useful information is suppressed in this statement, which gives no idea of how often the drug works or how similar the population they tested it on is to you. This makes it very difficult to decide what to do. A more useful statement would tell us the fraction of times a drug or procedure has worked on a patient with similar age and fitness level. Even in the cases when the doctors themselves don’t understand statistics, they can almost certainly provide some data or information.
In fairness, the heterogeneity of the population, with different individuals responding to drugs in different ways, makes determining how a medicine will work a complicated question. So let’s first consider a simpler case in which we can test on a single individual. Let’s use as an example the procedure for testing whether or not aspirin helps relieve your headache.
The way to figure this out seems pretty easy: take an aspirin and see if it works. But it