The Information - James Gleick [117]
When Licklider described some distortion as neither linear nor logarithmic but “halfway between,” Wiener interrupted.
“What does ‘halfway’ mean? X plus S over N?”
Licklider sighed. “Mathematicians are always doing that, taking me up on inexact statements.”♦ But he had no problem with the math and later offered an estimate for how much information—using Shannon’s new terminology—could be sent down a transmission line, given a certain bandwidth (5,000 cycles) and a certain signal-to-noise ratio (33 decibels), numbers that were realistic for commercial radio. “I think it appears that 100,000 bits of information can be transmitted through such a communication channel”—bits per second, he meant. That was a staggering number; by comparison, he calculated the rate of ordinary human speech this way: 10 phonemes per second, chosen from a vocabulary of 64 phonemes (26, “to make it easy”—the logarithm of the number of choices is 6), so a rate of 60 bits per second. “This assumes that the phonemes are all equally probable—”
“Yes!” interrupted Wiener.♦
“—and of course they are not.”
Wiener wondered whether anyone had tried a similar calculation for “compression for the eye,” for television. How much “real information” is necessary for intelligibility? Though he added, by the way: “I often wonder why people try to look at television.”
Margaret Mead had a different issue to raise. She did not want the group to forget that meaning can exist quite apart from phonemes and dictionary definitions. “If you talk about another kind of information,” she said, “if you are trying to communicate the fact that somebody is angry, what order of distortion might be introduced to take the anger out of a message that otherwise will carry exactly the same words?”♦
That evening Shannon took the floor. Never mind meaning, he said. He announced that, even though his topic was the redundancy of written English, he was not going to be interested in meaning at all.
He was talking about information as something transmitted from one point to another: “It might, for example, be a random sequence of digits, or it might be information for a guided missile or a television signal.”♦ What mattered was that he was going to represent the information source as a statistical process, generating messages with varying probabilities. He showed them the sample text strings he had used in The Mathematical Theory of Communication—which few of them had read—and described his “prediction experiment,” in which the subject guesses text letter by letter. He told them that English has a specific entropy, a quantity correlated with redundancy, and that he could use these experiments to compute the number. His listeners were fascinated—Wiener, in particular, thinking of his own “prediction theory.”
“My method has some parallelisms to this,” Wiener interrupted. “Excuse me for interrupting.”
There was a difference in emphasis between Shannon and Wiener. For Wiener, entropy was a measure of disorder; for Shannon, of uncertainty. Fundamentally, as they were realizing, these were the same. The more inherent order exists in a sample of English text—order in the form of statistical patterns, known consciously or unconsciously to speakers of the language—the more predictability there is, and in Shannon’s terms, the less information is conveyed by each subsequent letter. When the subject guesses the next letter with confidence, it is redundant, and the arrival of the letter contributes no new information. Information is surprise.
The others brimmed with questions about different languages, different prose styles, ideographic writing, and phonemes. One psychologist asked whether newspaper writing would look different, statistically, from the work of James Joyce. Leonard Savage, a statistician