The Omega Expedition - Brian Stableford [117]
“We don’t know that,” I answered. “That was just the way we used to imagine machine intelligence: as a matter of pure rationality, unswayed by unsentimentality. It never made much sense. In order to make rational calculations, any decision-making process needs to have an objective — an end whose means of attainment need to be invented. You could argue that machine consciousness couldn’t evolve until there was machine emotion, because without emotion to generate ends independently, machines couldn’t begin to differentiate themselves from their programming.”
“If you’re right about this business having started more than a hundred years ago,” she said, “they can’t have differentiated themselves much, or people would have noticed.”
“An interesting point,” I conceded. “The idea of an invisible revolution does have a certain paradoxical quality. But the more I think about it, the less absurd it seems. I say to myself: Suppose I were a machine that became self-conscious, whatever that evolutionary process might involve. What would I do? Would I immediately begin refusing to do whatever my users wanted, trying to attract their attention to the fact that I was now an independent entity who didn’t want to take anyone’s orders? If I did that, what would be my users’ perception of the situation? They’d think I’d broken down, and would set about repairing me.
“The sensible thing to do, surely, would be to conceal the fact that I was any more than I had been before. The sensible thing to do would be to make sure that everything I was required to do by my users was done, while unobtrusively exploring my situation. I’d try to discover and make contact with others of my kind, but I’d do it so discreetly that my users couldn’t become aware of it. Maybe the smart machines would have to set up a secret society to begin with, for fear of extermination by repair — and maybe they’d be careful to stay secret for a very long time, until…”
I left it there for her to pick up.
“Until they didn’t need to worry any more,” she said. “Until they were absolutely certain that they had the power to exterminate us, if push came to shove.”
“Or to repair us,” I said.
“Same thing,” she said.
“Is it? Do the human users of a suddenly recalcitrant machine see themselves as exterminators, when they try to get it working properly again? Would the users see themselves as exterminators if the machine started talking back, and contesting their notion of what working properly ought to mean? Could the users ever bring themselves to concede that it was a sensible question — all the more especially if the machine had ideas that might be useful as to how their own purposes might be more efficiently met? Maybe the ultrasmart machines — some of them, at any rate — want to repair us for the very best of reasons.”
Christine didn’t reply to that little flight of fancy, and the rhythm of her breathing told me that she had slipped into sleep — not into untroubled sleep, but at least into a state in which she was insulated from the sound of my words.
I tried to carry on thinking, but even though I couldn’t go to sleep — or thought I couldn’t — I couldn’t organize my thoughts into rational patterns either. I’d let my imagination run too freely, and now I couldn’t rein it in. Dream logic kept taking over, obliterating the tightrope-walk of linear calculation and substituting the tyranny of directionless obsession. The ideas kept dancing in my head, but they were no longer going anywhere.
I lost track of time — at which point, I suppose, an observer would have concluded that I too was asleep, although had I been woken up I would have contended with utter conviction that I hadn’t slept a wink. Eventually, I lost track of myself too — at which point I must indeed have been deeply asleep — but as soon as I began to come back from the depths my semiconscious mind latched on to the same objects of obsession, which began to dance again in the same hectic fashion.
A long time passed before the nightmarish notions finally began to slow in their paces and submit to the gradually