I'm Feeling Lucky_ The Confessions of Google Employee Number 59 - Douglas Edwards [91]
"Urs just said, 'We will have a 1B index,'" recalls Ben Gomes, "and it seemed like crazy talk." Gomes knew that increasing Google's size by that amount required more than slight improvements in current methods. "My advisor had a saying," he told me. "'An order of magnitude is qualitative, not quantitative.' When you go up by an order of magnitude, the problem is different enough that it demands different solutions. It's discontinuous."
Given the two equally impossible tasks—meeting Yahoo's requirements and creating the world's first billion-URL index—Larry and Sergey doubled down. Google would do both and do them at the same time. Google would begin answering Yahoo queries on Monday, July 3, leaving the Fourth of July holiday to repair any major disasters that might occur. It was a one-day margin of error in which to parse convoluted code, find bugs, squash them, and, if necessary, restart a system that had never dealt with the load it would now be called upon to carry twenty-four hours a day, seven days a week.
Before You Can Run, You Have to Crawl
Late on an April evening in the Googleplex, a steady clicking sound filled the space between the fabric walls, echoing the spring rain tapping against the windows. In his office, Jeff Dean stood looking over the shoulder of Sanjay Ghemawat, suggesting code variations as Sanjay typed and lines of text scrolled off his screen like the stairs of an ascending escalator.
Craig Silverstein drifted in, twisting a purple and black toy spider in his hands. Craig looked over Sanjay's other shoulder and set the spider down to point to a command about which he had a question. Jeff answered and Craig, satisfied, left. The spider remained—forsaken, but not alone. A block puzzle held together by an elastic band and a grip strengthener lay nearby, visible testaments to Craig's recurring visits. Ben Gomes stood in the hall, tossing beanbags into the air and catching some of them. Juggling rejuvenated him after hours of screen time and broke up the crusty patches that formed over his creativity. Inside the "Ben Pen," the office he shared with Ben Polk and Ben Smith,* the soundtrack of the 1986 film The Mission, a tale of pride and the struggle for redemption, swelled to a crescendo.
They were babysitting the crawl.
The engineers took turns monitoring the crawl's progress to make sure it didn't fail because of a single machine running amok. Urs had finally confided in his team why he was pushing so hard for so much new code. Everyone now knew the Yahoo deal was real and the deadline firm. Intensity set around the engineering group, a hardening cement of stress and pressure that grew firmer with each passing hour.
Even the implacable JeffnSanjay were not immune to its effects. "It was only a few months after I joined," recalls Sanjay, "and it was one of the most stressful times working at Google. We saw the deal and we knew when we had to get things done. We could do the math."
Sanjay worked on the new indexing system, which would be tested for the first time with the April index. Instead of taking three or four months, the index would have to be finished in one. "When something failed, we were on it," he told me. "people would just wake up every few hours and see if anything bad had happened, and then go fix it. The working style was long hours, constant attention, and quickly fixing things as they went bad."
Jeff remembers this entire period as the most demanding of his decade at Google: "From March 2000 to the end of 2001 was just frantically spent redesigning our systems and trying out different ideas on a very short time scale. The difficulty was the product of several different dimensions. How many queries do you have? How big is your index? How often do you want to update your index?"
The Bens provided