Flim-Flam! Psychics, ESP, Unicorns, and Other Delusions - James Randi [130]
"I have other experiments," said Schmidt, "that did not work out." How many, we were not told. The number of successes that resulted from the "good" experiments looked very small indeed. Professor Beloff of the University of Edinburgh had tried the same experiment with a random generator and had found nothing.
In the question-and-answer period following Schmidt's talk, the persistent problem of "optional stopping" came up. Briefly stated, if the subject is allowed to stop whenever he or she wants, there is no value to the experiment, since the subject can stop or be stopped when ahead, and the total result is a win, regardless of what would have happened had the test continued. For this reason, the experiment must have an announced number of trials determined firmly in advance, as was done. But optional stopping can also be optional continuing. It's the same problem. If results don't look too good—and remember that Schmidt's subjects had immediate feedback, telling them if they were ahead or not—it is easy to throw in another few dozen trials to see if we can get ahead before stopping.
Schmidt said he ran as many as 4 million "flips" of the generator in a test run to see if it was doing a proper job of fifty-fifty distribution. The machine seemed to be just fine—right on the mark. However, during the experiments, he said, the "flips" ran a small fraction either side of the midpoint line, as if a paranormal force were operating! But, as already mentioned, short runs tend to show greater deviance. Certainly the test run should have been of the same magnitude as the experimental run. As for the normal results of the very long test run, any tendency of the machine to produce a periodic bias in favor of either "heads" or "tails" would be effectively smoothed out. This assumes, of course, that fifty-fifty distribution is the only feature that the 4 million "flips" are used to test.
Also suspect was the practice of running a test series in advance to determine which way (positive or negative) the subject tended to lean that day, for if the machinery had any bias lasting a few thousand "flips," this would simply be testing for that bias, and following that with an official run that would only tend to exhibit that bias! The APS audience received assurances, however, that other random generators were periodically substituted during the tests. This seemed like starting a trip in a leaky balloon and carrying lots of repair materials, rather than taking off in a sound vehicle to begin with.
The bothersome thread that wound through all of Dr. Schmidt's talk was that basic defects were accounted for or excused by means of what seemed to be after-the-fact adjustments. When the experimental results showed nothing more than the law of averages, it was because "no enthusiasm" was applied. Negative results were reevaluated as positive ones after certain statistical ceremonies were carried out. There were aspects of the procedure that came to light only after close questioning. The telling moment of the evening came when Dr. Ray Hyman, whose investigations of parapsychological testing procedures have been extensive, asserted that Dr. Schmidt was using the science of statistics far beyond its normal limits and extracting from it much more than was justified by the exhibited results. There seemed to be considerable agreement on this point among the APS members present.
In his reply, Dr. Schmidt seemed to have thought of some means of getting around these objections. But I was most interested in another flaw. I asked him why he had not mentioned the most startling of all his observations, the one that stood out more than any other result ever claimed in the field.