Drunkard's Walk - Leonard Mlodinow [69]
Wine tasters are also often fooled by the flip side of the expectancy bias: a lack of context. Holding a chunk of horseradish under your nostril, you’d probably not mistake it for a clove of garlic, nor would you mistake a clove of garlic for, say, the inside of your sneaker. But if you sniff clear liquid scents, all bets are off. In the absence of context, there’s a good chance you’d mix the scents up. At least that’s what happened when two researchers presented experts with a series of sixteen random odors: the experts misidentified about 1 out of every 4 scents.21
Given all these reasons for skepticism, scientists designed ways to measure wine experts’ taste discrimination directly. One method is to use a wine triangle. It is not a physical triangle but a metaphor: each expert is given three wines, two of which are identical. The mission: to choose the odd sample. In a 1990 study, the experts identified the odd sample only two-thirds of the time, which means that in 1 out of 3 taste challenges these wine gurus couldn’t distinguish a pinot noir with, say, “an exuberant nose of wild strawberry, luscious blackberry, and raspberry,” from one with “the scent of distinctive dried plums, yellow cherries, and silky cassis.”22 In the same study an ensemble of experts was asked to rank a series of wines based on 12 components, such as alcohol content, the presence of tannins, sweetness, and fruitiness. The experts disagreed significantly on 9 of the 12 components. Finally, when asked to match wines with the descriptions provided by other experts, the subjects were correct only 70 percent of the time.
Wine critics are conscious of all these difficulties. “On many levels…[the ratings system] is nonsensical,” says the editor of Wine and Spirits Magazine.23 And according to a former editor of Wine Enthusiast, “The deeper you get into this the more you realize how misguided and misleading this all is.”24 Yet the rating system thrives. Why? The critics found that when they attempted to encapsulate wine quality with a system of stars or simple verbal descriptors such as good, bad, and maybe ugly, their opinions were unconvincing. But when they used numbers, shoppers worshipped their pronouncements. Numerical ratings, though dubious, make buyers confident that they can pick the golden needle (or the silver one, depending on their budget) from the haystack of wine varieties, makers, and vintages.
If a wine—or an essay—truly admits some measure of quality that can be summarized by a number, a theory of measurement must address two key issues: How do we determine that number from a series of varying measurements? And given a limited set of measurements, how can we assess the probability that our determination is correct? We now turn to these questions, for whether the source of data is objective or subjective, their answers are the goal of the theory of measurement.
THE KEY to understanding measurement is understanding the nature of the variation in data caused by random error. Suppose we offer a number of wines to fifteen critics or we offer the wines to one critic repeatedly on different days or we do both. We can neatly summarize the opinions employing the average, or mean, of the ratings. But it is not just the mean that matters: if all fifteen critics agree that the wine is a 90, that sends one message; if the critics produce the ratings 80, 81, 82, 87, 89, 89, 90, 90, 90, 91, 91, 94, 97, 99, and 100, that sends another. Both sets of data have the same mean, but they differ in the amount they vary from that mean. Since the manner in which data points are distributed is such an important piece of information, mathematicians created a numerical measure of variation to describe it. That number is called the sample standard deviation. Mathematicians also measure the variation by its square, which is called the sample variance.
The sample standard