Reinventing Discovery_ The New Era of Networked Science - Michael Nielsen [52]
Open Data
What’s made the SDSS such a success? We’ve already discussed some of the reasons: the SDSS has an excellent telescope and broad coverage of the sky. But those can’t be the only reasons. In the 1940s and 1950s astronomers used the giant 5-meter Palomar telescope outside San Diego, California, to carry out the Palomar Observatory Sky Survey. But while the Palomar telescope is in some respects even better than the SDSS telescope, the Palomar survey had a much less dramatic impact on astronomy. Why is that the case? The main reason is that the Palomar survey produced bulky photographic plates, which are expensive to move around and to duplicate, and so could only be accessed by a few people. By contrast, the SDSS has used the internet to share its data with the entire worldwide community of astronomers. Since 2001, the SDSS has done seven major data releases, putting its images (and other data) up on the web where anyone can download them. If you want, you can go right now to the SDSS’s online SkyServer, and download stunning images of distant galaxies. Anyone can do it, and the site is designed to be used not just by professional astronomers, but also by members of the general public. The tools on the site range from tours of the most beautiful sights in the sky, through to the ability to send sophisticated database queries that will return images with particular desired characteristics. The site even contains tutorials explaining how to do things like find asteroids, or star-forming regions in other galaxies.
This open sharing of data by the SDSS seems like a small innovation when compared to the radical approaches to collective intelligence we saw in examples such as the Polymath Project and Kasparov versus the World. But the impact of the open sharing of data by the SDSS is enormous. It means that people such as Todd Boroson and Tod Lauer—people who aren’t members of the SDSS collaboration—can come along and ask fundamental questions that no one had ever thought to ask before: “Can we use the SDSS data to search for pairs of orbiting black holes?” In science, discoveries are often constrained by the limits to our knowledge. But experiments such as the SDSS produce such an extraordinary wealth of knowledge—more than 70 terabytes of data, far beyond the ability of any single human to comprehend—that they confound that expectation. Confronted by such a wealth of data, in many ways we are not so much knowledge-limited as we are question-limited. We’re limited by our ability to ask the most ingenious and outrageous and creative questions. By opening up its data to the whole world, the SDSS has enabled people such as Boroson and Lauer to ask such questions, questions that might never have been asked if access to the data was limited. It’s the same thing we saw in Swanson’s discovery of the migraine-magnesium connection: Swanson used no facts that weren’t already known, but by asking a new question of existing knowledge, he made a valuable discovery. It’s a variation on the designed serendipity we saw in part 1. Instead of broadcasting a question to the world and hoping for an answer, projects such as the SDSS broadcast data to the world, in the belief that people will ask unanticipated questions that lead to new discoveries.
The SDSS’s sharing of data isn’t just important because of the discoveries it enables. It’s also important because sharing data in this way, as simple and obvious as it might seem, in fact is a radically daring step for the scientists involved. Most scientists guard their data jealously. Their data is their raw record of experimental observations, and may lead to important new discoveries. It’s their special edge over their colleagues and competitors. Unusual as it may be for them to openly reveal their data, it’s even