I'm Feeling Lucky_ The Confessions of Google Employee Number 59 - Douglas Edwards [88]
That wasn't the case with Yahoo, where Udi Manber, a search specialist, was chief scientist.* "Udi wrote the contract," Urs told me, "so he paid attention to the important things." Manber would settle for nothing less than the best that Google could produce, which he knew to be more than what the search engine was offering at the time.
Larry and Sergey committed to Yahoo that Google would make numerous improvements in a matter of weeks: to set up two new data centers—including one on the East Coast—to freshen the index by crawling more frequently, to reduce spam in results, and to meet strict limits on latency, the time between the search being entered and the results being delivered. Each of these promises would require enormous effort to fulfill, but if all were met, Google would give Yahoo users a visibly better search experience.
Better, but not the best. "What was important," Urs confided to me, "was that if you syndicated something and you had your own property, you wanted to make sure that over time you could innovate and actually have something better than Yahoo search powered by Google. And that was Google search, unrestricted by commercial agreement."
Had the company bitten off more than it could chew? Yahoo's traffic dwarfed Google's,† and the moment Inktomi stopped answering Yahoo queries, Google would need to respond to each and every one of them quickly and correctly. With the capacity we had in place at the time, it would be like hooking a garden hose to a fire hydrant.
"Operationally it was a large buildup," Urs said, "so we needed to get lots of servers." Five thousand, to be exact, each of which required hand assembly. The new data centers that would hold them had yet to be identified or prepped for occupancy. New networking systems would have to be developed to ensure queries went to the right place and results were identical regardless of where the responding machine resided.
The Google operations team* worked with the contractor, Rackable Systems,† building the machines from parts Gerald supplied. A "bat cave" was set up between Charlie's café and the marketing department for the ops staff to test and burn in servers before shipping them off to the East Coast. The loaded racks were large and heavy but fragile. Google tried renting a truck and having ops move the racks themselves, but one fell over in transit and almost crushed a technician. The decision was made to splurge on professionals. That didn't stop the ops guys from tossing hundreds of thousands of dollars' worth of RAM into their cars and driving it all over town.
Once all the racks were ready, they needed to be moved cross-country practically overnight. Christopher Bosch hired a truck to drive nonstop to the new data center in Virginia. It would leave the highway only to change drivers en route, and once it reached the data center, the racks would roll out the back and directly into the new server farm.
Serving capacity was just the first item on a very long checklist, and in some ways the easiest. The pressure to build quickly was enormous, but at least with hardware the task was clearly defined, the process known, and the progress clearly measurable. Adding hardware to add capacity would not solve Google's problems by itself, however, even without the Yahoo deal.
"Our traffic was increasing eight percent a week for a long time," said Jeff Dean. "Any time you have that rate of growth, you basically have to make software improvements continuously because you can't get the hardware deployed fast enough. They were working as hard as they could, but if you add four percent machines per week, and you've got eight percent growth, that's not good. So we continuously worked on improving our serving system and looked at alternative designs for pieces of it to get more throughput." That kind of software required creativity and design breakthroughs that could not be scheduled in