• Choose a category
• All
• Classic-Fiction

By Root 395 0

games like checkers, chess, and go, or efforts to train computers to understand human speech. Data-driven intelligence can be applied to these traditionally human tasks—it can understand human speech, or play chess—but where it really excels is in solving different kinds of problems, problems involving skills complementary to human intelligence, problems such as Swanson’s searches of the medical research literature, or Boroson and Lauer’s mining of the SDSS data for pairs of orbiting black holes. A full-fledged data-driven intelligence would be able to play checkers, chess, or go, but it wouldn’t play them for fun. It would play games with a scope whose complexity was entirely beyond human comprehension.

The term “intelligence” is often used to mean some kind of generalized intellectual ability. Data-driven intelligence is more targeted in nature, with different kinds of data-driven intelligence used to solve different kinds of problems. We’ll see an explicit example in the next section, which looks at the algorithms biologists use to do genome sequencing. A quite different set of algorithms is used to do searching in services such as Medline. For each problem, a different kind of data-driven intelligence is required. A consequence is that data-driven intelligence in some problem domain may start out quite stupid, but gradually get smarter as we develop improved methods. For instance, when Swanson did his migraine-magnesium work, search tools such as Medline used relatively simple ideas. Today’s search engines use much more sophisticated ideas, and tomorrow’s search engines will no doubt be much better still. Indeed, as data-driven intelligence helps companies such as Google turn a profit, those companies pour money into developing still better techniques, resulting in a virtuous circle of improvement.

How is data-driven intelligence related to collective intelligence? Actually, that’s not quite the right question for our discussion. We’re interested in data-driven intelligence as a way of augmenting our own intelligence, and so a better question is: how does data-driven intelligence relate to the tools we studied in part 1, the tools that amplify collective intelligence? As we saw, those tools work by restructuring expert attention so it’s more effectively allocated. There’s thus no direct relationship between tools that amplify collective intelligence and data-driven intelligence. But the two can be used in a complementary way. For instance, we’ve seen how data-driven tools such as Medline provide new ways of finding meaning hidden in the collective knowledge of large groups of people, such as the biomedical community. And data-driven tools such as Google can be used to amplify our collective intelligence by helping us find the information and the people that we should be paying attention to. Conversely, Google uses our collective intelligence to build its service, mining the web for content, and using the link structure of the web to figure out which pages are most important. So even though data-driven and collective intelligence are different, they can be used to reinforce each other.

This is not a textbook on data-driven intelligence, and I won’t describe the hundreds of clever algorithms in use or under development. For us, data-driven intelligence is primarily important as a concept that unifies examples such as Google Flu Trends, the Sloan Great Wall, and Swanson’s migraine-magnesium discovery. Underlying all these examples are clever algorithms that extract meaning from data that is otherwise beyond human ality to comprehend. Data-driven intelligence is in some sense complementary to the data web: although data-driven intelligence can be applied to any data source, it reaches its fullest potential when applied to the richest possible data sources, and the data web is the richest data source we can imagine. Data-driven intelligence is what will allow us to take all the world’s knowledge and make meaning from it.

Data-Driven Intelligence in Biology

To make data-driven intelligence more concrete, let me describe