I'm Feeling Lucky_ The Confessions of Google Employee Number 59 - Douglas Edwards [14]
The overwhelming impression, as Jim led us past cage after cage of cooled processing power, was of fetishistic efficiency. Clean, pristine, and smoothly sculpted, these were more than machines, they were totems of the Internet economy. Here was eBay. Here Yahoo. Here Inktomi. Welcome to Stonehenge for the Information Age.
The common design element seemed to be a mechanized monolith centered in each cage, surrounded by ample space to set up a desk and a few chairs, with enough room left over for a small party of proto-humans to dance about beating their chests and throwing slide rules into the air.
At last we arrived at Google's cage. Less than six hundred square feet, it felt like a shotgun shack blighting a neighborhood of gated mansions. Every square inch was crammed with racks bristling with stripped-down CPUs. There were twenty-one racks and more than fifteen hundred machines, each sprouting cables like Play-Doh pushed through a spaghetti press. Where other cages were right-angled and inorganic, Google's swarmed with life, a giant termite mound dense with frenetic activity and intersecting curves. Narrow aisles ran between the rows of cabinets, providing barely enough space to pass if you didn't mind shredding clothes and skin on projecting screws and metal shards.
It was improbably hot after our stroll through a freezer to get there, and we were soon sweating and shedding outerwear. On the floor, sixteen-inch metal fans vibrated and vainly pushed back against the heat seeping out from the racks around us—their feeble force doing little more than raise the temperature of Inktomi's adjacent cage by a few degrees.
We went to work. First the ops team attached Panduit cable troughs to the sides of the cabinets with adhesive tape. Then we began gently placing the free-hanging cables in the troughs and twist-tying them together so they no longer draped over the face of the machines like the bangs of a Harajuku Girl.
I tackled the rack labeled "U." It has long since been retired, but I like to think that those user queries routed to U got their answers a nanosecond or two faster because of my careful combing of the cables.
Why, you might ask, did Google do things this way? In addition to the efficiency gained by running cheap, redundant servers, Google was exploiting a loophole in the laws of co-lo economics. Exodus, like most hosting centers, charged tenants by the square foot. So Inktomi paid the same amount for hosting fifty servers as Google paid for hosting fifteen hundred. And the kicker? Power, which becomes surprisingly expensive when you gulp enough to light a neighborhood, was included in the rent. When Urs renegotiated the lease with Exodus, Jim spelled out exactly how much power he needed. Not the eight twenty-amp circuits normally allocated to a cage the size of Google's; he wanted fifty-six.
"You just want that in case there's a spike, right?" asked the Exodus sales rep with a look of surprise. "There's no way you really need that much power for a cage that size."
"No," Jim told him. "I really need all fifty-six to run our machines."
It's rumored that at one point Google's power consumption exceeded Exodus's projections fifty times over.* It didn't help that Google sometimes started all of its machines at once, which blew circuit breakers left and right until Google instituted five-second delays to keep from burning down the house.
Air-conditioning came standard, too. Again, Exodus based their calculations on a reasonability curve. No reasonable company would cram fifteen hundred micro-blast furnaces into a single cage, because that would require installing a separate A/C unit. Google did. We were a high-maintenance client.
CableFest '99 was the one and only time I entered a Google data center. It gave me an appreciation of the magnitude of what we were building and how differently we were doing it. I can't say it inspired confidence to lay my untrained hands on our cheap little generic servers, lying open to the controlled elements on crumbly