• Choose a category
• All
• Classic-Fiction

In other words, while lots of people wanted to read comments about other people’s papers, almost no one wanted to actually write comments.

The Nature trial is just one of many attempts to build user-contributed comment sites for science. Physics, in particular, has seen many such sites, perhaps because it was the first field to broadly adopt the web as a way of distributing scientific papers. The first attempt was the site Quick Reviews, which came online in 1997, and was discontinued for lack of use in 1998. A similar site, Physics Comments, was built a few years later, but suffered the same fate, being discontinued in 2006. A still more recent site, Science Advisor, is still active, but has more members (1,240) than reviews (1,119) as I write. It seems that many scientists want to read comments on scientific papers, but very few want to volunteer to write such comments.

Why are the user-contributed comment sites failing? In principle, most scientists agree that it would be tremendously useful if thoughtful commentary on scientific papers was widely available. But if that’s true, then it seems like a puzzle that these sites—many of them well designed and well supported—fail, when the comment sections on sites such as Amazon.com thrive. The problem the scientific comment sites have is that while thoughtful commentary on scientific papers is tremendously useful for other scientists, that doesn’t mean it’s in anyone’s individual best interest to write comments. Imagine how things look from the point of view of an individual scientist considering commenting on such a site. Why write a comment when you could be doing something more useful to you individually, like writing a paper or a grant? Even if you did write a comment, you’d likely be reluctant to publicly criticize someone else’s paper. After all, the person you criticize might be an anonymous referee in a position to scuttle your next paper or grant application.

The contrast between the failures of the user-contributed comment sites for science and the success of the Amazon.com reviews is stark. To pick just one example, you’ll find more than 1,500 reviews of Pokemon products at Amazon.com, more than the total number of reviews on all the science comment sites I described above. You may object that there are more people who buy Pokemon products than there are scientists. That’s true. But there are still more than a million professional scientists in the world, and those scientists spend much of their working lives forming opinions of papers written by others, far more time than even the most enthusiastic parents can spend on Pokemon. It’s a ludicrous situation: popular culture is open enough that people feel a desire to write Pokemon reviews, yet scientific culture is so closed that scientists won’t publicly share their opinions of scientific papers in an analogous way. Some people find this contrast curious or amusing; I believe it signifies something seriously amiss with science.

The Modern Challenge for Open Science

The failure of the science wikis and the user-contributed comment sites for science is part of a much larger pattern. Projects such as the Polymath Project, Galaxy Zoo, and Foldit have all been very successful, but that success has come in part because of a fundamental conservatism: all of them ultimately aim to produce scientific papers. Tools such as the science wikis and user-contributed comment sites break away from this conservatism, since contributions to such sites are ends in themselves, and don’t directly result in scientific papers. Unfortunately, the result is that career-minded scientists have little incentive to contribute to such sites, and instead focus their efforts on doing what is rewarded: writing papers. The grand ideas for amplifying collective intelligence that we discussed in part 1 have little chance to thrive when incremental ideas such as science wikis and user-contributed comment sites are already beyond the pale. Many of the tools with the potential to most dramatically