I'm Feeling Lucky_ The Confessions of Google Employee Number 59 - Douglas Edwards [90]
The old crawler would never be able to jump through Yahoo's hoops.
1B Wannabe
Urs recognized from the beginning that before Google could make a quantum leap to a higher state it would have to correct the mistakes of the past—especially the creaking codebase underlying the main systems. "We've fixed some of the problems," he noted after putting out the fires immediately threatening to consume Google the day he first walked in the door, "but we should really restart completely from scratch." That was a risky thing to do, because it required using resources that were in short supply to fix something that wasn't yet broken. Most companies would put off such a complex task until a time they were less overcommitted.
"This wasn't the burning problem of the day," Urs told me. "The site wasn't down because of it; it was just a productivity problem. If you stayed in the old, messy world too long, your effectiveness would continue to go down." He gave the green light in the fall of 1999 to create a new codebase called Google Two. New systems would run on Google Two, and the original codebase would be phased out. Jeff and Craig started working on it, but writing new infrastructure took time—and time refused to stand still, even for the engineers at Google.
In the months that followed, ballooning traffic increased the pressure at every point, and as Urs had predicted, cracks appeared in the lines of the original Stanford code.
Sitting in the hot tub at Squaw Valley Resort during the company's annual ski trip, two months before the March index meltdown, Craig suggested to Jeff that they write an entirely new crawler and indexer for Google Two. It would be cleaner than replacing the old ones bit by bit. Jeff saw the logic in that. The MarIndex suddenly gave that project urgency. Joined by Sanjay and Ben Gomes, they ripped out Google's aging guts and replaced them with a streamlined block of high-efficiency algorithms.
The team didn't know Yahoo was floating out there, nibbling on Omid's line, or that when he landed the contract in May, they would have only a month to complete their work. The new systems would have to be completely stable under triple the highest load Google had ever handled. They would need to distribute queries to thousands of servers in multiple data centers and automatically balance the flux of traffic on the basis of machine availability. The engineers couldn't shut off Google while they tested the system, and they couldn't drop a single Yahoo query once the deal was done. Google embraced risk, but sensible, talented engineers could infer that this indicated a company-wide death wish. So many ways to go so horribly wrong. Urs evaluated the situation and his team's capabilities and decided they needed more of a challenge.
The great white whale of search in early 2000 was an index of a billion URLs. No one had come within sight of anything close, but Urs set out to harpoon just such a beast with Google's shiny new crawler. While only half the pages in the index would be fully indexed (meaning the crawler would examine their full content, not just identify the URL at which the content lived), it would still overshadow Inktomi, which claimed to have crawled 110 million full pages. A billion URLs were not required for the Yahoo deal, but to the crawl team, the "1B" index held epic significance. It would catapult Google into the undisputed lead as the builder of the best, most scalable search technology in the world.
"In search," Urs believed, "the discussion