Everything Is Obvious_ _Once You Know the Answer - Duncan J. Watts [86]
PREDICTING THE PRESENT
At a higher level, the Web as a whole can also be viewed as a form of crowdsourcing. Hundreds of millions of people are increasingly turning to search engines for information and research, spending ever more time browsing news, entertainment, shopping, and travel sites, and increasingly sharing content and information with their friends via social networking sites like Facebook and Twitter. In principle, therefore, one might be able to aggregate all this activity to form a real-time picture of the world as viewed through the interests, concerns, and intentions of the global population of Internet users. By counting the number of searches for influenza-related terms like “flu,” and “flu shots,” for example, researchers at Google and Yahoo! have been able to estimate influenza caseloads remarkably close to those reported by the CDC.11 Facebook, meanwhile, publishes a “gross national happiness” index based on users’ status updates,12 while Yahoo! compiles an annual list of most-searched-for items that serves as a rough guide to the cultural zeitgeist.13 In the near future, no doubt, it will be possible to combine search and update data, along with tweets on Twitter, check-ins on Foursquare, and many other sources to develop more specific indices associated with real estate or auto sales or hotel vacancy rates—not just nationally, but down to the local level.14
Once properly developed and calibrated, Web-based indices such as these could enable businesses and governments alike to measure and react to the preferences and moods of their respective audiences—what Google’s chief economist Hal Varian calls “predicting the present.” In some cases, in fact, it may even be possible to use the crowd to make predictions about the near future. Consumers contemplating buying a new camera, for example, may search to compare models. Moviegoers may search to determine the opening date of an upcoming film or to locate cinemas showing it. And individuals planning a vacation may search for places of interest and look up airline costs or price hotel rooms. If so, it follows that by aggregating counts of search queries related to retail activity, moviegoing, or travel, one might be able to make near-term predictions about behavior of economic, cultural, or political interest.
Determining what kind of behavior can be predicted using searches, as well as the accuracy of such predictions and the timescale over which predictions can be usefully made are therefore all questions that researchers are beginning to address. For example, my colleagues at Yahoo! and I recently studied the usefulness of search-query volume to predict the opening weekend box office revenues of feature films, the first-month sales of newly released video games, and the Billboard “Hot 100” ranking of popular songs. All these predictions were made at most a few weeks in advance of the event itself, so we are not talking about long-term predictions here—as discussed in the previous chapter, those are much harder to make. Nevertheless, even having a slightly better idea a week in advance of audience interest might help a movie studio or a distributor decide how many screens to devote to which movies in different local regions.15
What we found is that the improvement one can get from search queries over other types of public data—like production budgets or distribution plans—is small but significant. As I discussed in the last chapter, simple models based on historical data are surprisingly hard to outperform, and the same rule applies to search-related data as well. But there are still plenty of ways in which search and other Web-based data could help with predictions. Sometimes, for example, you won’t have access to reliable sources of historical data—say you’re launching a new game that isn’t like games you’ve launched in the past, or because you don’t have access to a competitor’s sales figures. And sometimes, as I’ve also discussed, the future is not like the past—such as when normally placid economic indicators suddenly increase in volatility or