The Information - James Gleick [96]
The theorem was an extension of Nyquist’s formula, and it could be expressed in words: the most information that can be transmitted in any given time is proportional to the available frequency range (he did not yet use the term bandwidth). Hartley was bringing into the open a set of ideas and assumptions that were becoming part of the unconscious culture of electrical engineering, and the culture of Bell Labs especially. First was the idea of information itself. He needed to pin a butterfly to the board. “As commonly used,” he said, “information is a very elastic term.”♦ It is the stuff of communication—which, in turn, can be direct speech, writing, or anything else. Communication takes place by means of symbols—Hartley cited for example “words” and “dots and dashes.” The symbols, by common agreement, convey “meaning.” So far, this was one slippery concept after another. If the goal was to “eliminate the psychological factors involved” and to establish a measure “in terms of purely physical quantities,” Hartley needed something definite and countable. He began by counting symbols—never mind what they meant. Any transmission contained a countable number of symbols. Each symbol represented a choice; each was selected from a certain set of possible symbols—an alphabet, for example—and the number of possibilities, too, was countable. The number of possible words is not so easy to count, but even in ordinary language, each word represents a selection from a set of possibilities:
For example, in the sentence, “Apples are red,” the first word eliminated other kinds of fruit and all other objects in general. The second directs attention to some property or condition of apples, and the third eliminates other possible colors.…
The number of symbols available at any one selection obviously varies widely with the type of symbols used, with the particular communicators and with the degree of previous understanding existing between them.♦
Hartley had to admit that some symbols might convey more information, as the word was commonly understood, than others. “For example, the single word ‘yes’ or ‘no,’ when coming at the end of a protracted discussion, may have an extraordinarily great significance.” His listeners could think of their own examples. But the point was to subtract human knowledge from the equation. Telegraphs and telephones are, after all, stupid.
It seemed intuitively clear that the amount of information should be proportional to the number of symbols: twice as many symbols, twice as much information. But a dot or dash—a symbol in a set with just two members—carries less information than a letter of the alphabet and much less information than a word chosen from a thousand-word dictionary. The more possible symbols, the more information each selection carries. How much more? The equation, as Hartley wrote it, was this:
H = n log s
where H is the amount of information, n is the number of symbols transmitted, and s is the size of the alphabet. In a dot-dash system, s is just 2. A single Chinese character carries so much more weight than a Morse dot or dash; it is so much more valuable. In a system with a symbol for every word in a thousand-word dictionary, s would be 1,000.
The amount of information is not proportional to the alphabet size, however. That relationship is logarithmic: to double the amount of information, it is necessary to quadruple the alphabet size. Hartley illustrated this in terms of a printing telegraph—one of the hodgepodge of devices, from obsolete to newfangled, being hooked up to electrical circuits. Such telegraphs used keypads arranged according to a system devised in France by Émile Baudot. The human operators