Everything Is Obvious_ _Once You Know the Answer - Duncan J. Watts [62]
All three predictions were made by authoritative-sounding sources and were explicit enough to have been scored for accuracy. But how accurate were they? To be honest, I have no idea. The New York Times doesn’t publish statistics on the accuracy of the predictions made in its pages, nor do most of the research companies that provide them. One of the strange things about predictions, in fact, is that our eagerness to make pronouncements about the future is matched only by our reluctance to be held accountable for the predictions we make. In the mid-1980s, the psychologist Philip Tetlock noticed exactly this pattern among political experts of the day. Determined to make them put their proverbial money where their mouths were, Tetlock designed a remarkable test that was to unfold over twenty years. To begin with, he convinced 284 political experts to make nearly a hundred predictions each about a variety of possible future events, ranging from the outcomes of specific elections to the likelihood that two nations would engage in armed conflict with each other. For each of these predictions, Tetlock insisted that the experts specify which of two outcomes they expected and also assign a probability to their prediction. He did so in a way that confident predictions scored more points when correct, but also lost more points when mistaken. With those predictions in hand, he then sat back and waited for the events themselves to play out. Twenty years later, he published his results, and what he found was striking: Although the experts performed slightly better than random guessing, they did not perform as well as even a minimally sophisticated statistical model. Even more surprisingly, the experts did slightly better when operating outside their area of expertise than within it.2
Tetlock’s results are often interpreted as demonstrating the fatuousness of so-called experts, and no doubt there’s some truth to that. But although experts are probably no better than the rest of us at making predictions, they are also probably no worse. When I was young, for example, many people believed that the future would be filled with flying cars, orbiting space cities, and endless free time. Instead, we drive internal combustion cars on crumbling, congested freeways; endure endless cuts in airplane service, and work more hours than ever. Meanwhile, Web search, mobile phones, and online shopping—the technologies that have, in fact, affected our lives—came more or less out of nowhere. Around the same time that Tetlock was beginning his experiment, in fact, a management scientist named Steven Schnaars tried to quantify the accuracy of technology-trend predictions by combing through a large collection of books, magazines, and industry reports, and recording hundreds of predictions that had been made during the 1970s. He concluded that roughly 80 percent of all predictions were wrong, whether they were made by experts or not.3
Nor is it just forecasters of long-term social and technology trends that have lousy records. Publishers, producers, and marketers—experienced and motivated professionals in business with plenty of skin in the game—have just as much difficulty predicting which books, movies, and products will become the next big hit as political experts have in predicting the next revolution. In fact, the history of cultural markets is crowded with examples of future blockbusters—Elvis, Star Wars, Seinfeld, Harry Potter, American Idol—that publishers and movie studios left for dead while simultaneously betting big on total failures.4 And whether we consider the most spectacular business meltdowns of recent times—Long-Term Capital Management in 1998, Enron in 2001, WorldCom in 2002, the near-collapse of the entire financial system in 2008