Beautiful Code [263]
Each worker process installs a signal handler that catches segmentation violations and bus errors. Before invoking a user Map or Reduce operation, the MapReduce library stores the sequence number of the record in a global variable. If the user code generates a signal, the signal handler sends a "last gasp" UDP packet that contains the sequence number to the MapReduce master. When the master has seen more than one failure on a particular record, it indicates that the record should be skipped when it issues the next re-execution of the corresponding Map or Reduce task.
A number of other extensions are discussed in a lengthier paper about MapReduce (see "Further Reading," below).
Distributed Programming with MapReduce > Conclusion
23.6. Conclusion
MapReduce has proven to be a valuable tool at Google. As of early 2007, we have more than 6,000 distinct programs written using the MapReduce programming model, and run more than 35,000 MapReduce jobs per day, processing about 8 petabytes of input data per day (a sustained rate of about 100 gigabytes per second). Although we originally developed the MapReduce programming model as part of our efforts to rewrite the indexing system for our web search product, it has shown itself to be useful across a very broad range of problems, including machine learning, statistical machine translation, log analysis, information retrieval experimentation, and general large-scale data processing and computation tasks.
Distributed Programming with MapReduce > Further Reading
23.7. Further Reading
A more detailed description of MapReduce appeared in the OSDI '04 conference:
"MapReduce: Simplified Data Processing on Large Clusters." Jeffrey Dean and Sanjay Ghemawat. Appeared in OSDI '04: Sixth Symposium on Operating System Design and Implementation, San Francisco, CA, December, 2004. Available from http://labs.google.com/papers/mapreduce.html.
A paper about the design and implementation of the Google File System appeared in the SOSP '03 conference:
"The Google File System." Sanjay Ghemawat, Howard Gobioff, and Shun-Tak Leung. 19th ACM Symposium on Operating Systems Principles, Lake George, NY, October, 2003. Available from http://labs.google.com/papers/gfs.html.
A paper describing the general hardware infrastructure at Google appeared in IEEE Micro:
"Web Search for a Planet: The Google Cluster Architecture." Luiz Barroso, Jeffrey Dean, and Urs Hoelzle. IEEE Micro, Volume 23, Issue 2 (March 2003), pp. 22–28. Available from http://labs.google.com/papers/googlecluster.html.
A language called Sawzall developed at Google for logs analysis runs on top of Map-Reduce:
"Interpreting the Data: Parallel Analysis with Sawzall." Rob Pike, Sean Dorward, Robert Griesemer, Sean Quinlan. Scientific Programming Journal Special Issue on Grids and Worldwide Computing Programming Models and Infrastructure 13:4, pp. 227– 298. Available from http://labs.google.com/papers/sawzall.html.
Distributed Programming with MapReduce > Acknowledgments
23.8. Acknowledgments
A number of people have made substantial contributions to the continued development and improvement of MapReduce, including Tom Annau, Matt Austern, Chris Colohan, Frank Dabek, Walt Drummond, Xianping Ge, Victoria Gilbert, Shan Lei, Josh Levenberg, Nahush Mahajan, Greg Malewicz, Russell Power, Will Robinson, Ioannis Tsoukalidis, and Jerry Zhao. MapReduce builds on a number of pieces of infrastructure developed at Google, including the Google File System and our cluster scheduling system. We would like to especially thank the developers