Online Book Reader

Home Category

I'm Feeling Lucky_ The Confessions of Google Employee Number 59 - Douglas Edwards [85]

By Root 1911 0
"Do you think you can fix yourself?"

The facilities team, for reasons known only to them, had unscrewed the support for the crossbeam while removing the wall. In Jim's professional medical opinion, the beam would have done him considerably more damage if it had landed a couple of inches to either side. "They were very, very, very kind to us after that," he said about the Exodus crew.

Jim finally found some unoccupied space in a corner of the building and Exodus agreed to throw fence walls around it. He spent the better part of June and July installing two thousand brand-new computers into the cage. The machines didn't always work. They were built quickly and with parts purchased at very reasonable prices. "For some racks," Jim recalls, "we got fifty-six out of eighty working, so we'd spend a week installing these machines and then another week repairing the ones that didn't install."

Eventually Larry let Jim in on why his work was so urgent. Google had signed a deal with Netscape to be their fall-through search engine. If Netscape users couldn't find what they wanted using Netscape's open directory, they would be able to search from the directory page using Google. So Google needed more computing power to handle the potential traffic.

Jim load-tested Google's capacity as he added machines—checking to see that it could handle the increased traffic and any occasional spikes that might occur. "In general," he told me, "you like to see two times capacity. For peaks, you like to see four times capacity. Netscape anticipated a one-point-seven-times increase over existing Google traffic, so I tested it. At nearly five times, we were completely in the clear."

It wasn't easy. Jim and Schwim were still at the data center installing machines the night of June 24. Netscape would announce the deal and start directing traffic to Google the morning of June 25. Schwim worked until two a.m., when the cumulative lack of sleep caught up with him and he went home to crash.

Fortunately, Jim had recruited another tech guy to help them over the finish line. Though it had been a while since Sergey had dirtied his hands installing machines, he stayed at the data center with Jim until five a.m. "He didn't know all the technical details of how the routing went," Jim remembered, "but he was in there crawling under the floorboards, running cables, and hooking up switches."

For Jim it was the culmination of weeks of exhausting physical labor, and when he finally dragged himself off to bed it was with a sense of accomplishment. Google had averted a potential disaster by tripling its capacity in record time.

An hour later, his phone rang. It was Sergey. "Get in here right away. We're melting down."

Netscape's press release had hit the newswires at six a.m. West Coast time. Within seconds Google's traffic had increased not the expected one-point-seven times, but sevenfold. The servers couldn't handle the load. Sergey and Jim rushed back to Exodus and began desperately throwing the last batch of machines they had into racks and hooking them up.

Meanwhile they did everything they could to clear away extraneous demands on Google's infrastructure. They stopped the crawler from adding websites to Google's index and reallocated those machines to serving results. It helped, but not enough. Response times had slowed perceptibly, and some users got no results at all. Google's most important launch to date teetered on the brink of becoming an epic pooch-screwing.

The atmosphere in the office Craig Silverstein shared with Amit Patel was grim as Larry and Sergey, Urs, and the rest of Google's engineers reviewed their options. Netscape was not a small partner like their first client, VMWare. If this relationship went down the tubes, everyone would know and Google's tech reputation would be toast. They could think of only one way to increase capacity to handle Netscape's users.

"Shut off queries to Google.com," Larry instructed the team.

For the next couple of hours, anyone who went to Google.com saw a static page explaining that Google was down. Every computer

Return Main Page Previous Page Next Page

®Online Book Reader