Final Jeopardy (Alexandra Cooper Mysteries) - Linda Fairstein [43]
From the very beginning, Brown kept tight control of the Jeopardy data. He distributed two thousand clues at a time, which the team used to train Blue J. The risk they faced, as in any statistical analysis, was that they’d fine-tune the machine too precisely to each batch of questions. This tendency to model too closely to a training set is known as overfitting, and it’s a serious problem.
Anyone who has ever studied a foreign language knows all about it. Students inevitably overfit to the French or Spanish or Mandarin that their teacher speaks. They adjust to her rhythms and syntax and come to associate that single voice with the language itself. A trip to Paris or Beijing often brings a rude awakening. In Blue J’s education, each training set was a single teacher. When the computer started to score well on a training set, the researchers would test it on Jeopardy clues it had never seen before. This was a blind set of data, a few thousand clues that no one but Brown had seen. Each time Blue J ventured from its comfortable clues into an unfamiliar set of data, its results would drop about 5 percent. But still, its overall scores were rising. Brown would release another training set, and the process would start over.
The broader question, naturally, was whether the Jeopardy challenge itself was one giant exercise in overfitting. Jeopardy, in a sense, was a single training set of 185,000 clues, including general knowledge and a mix (that Ferrucci’s team quickly quantified) of puzzles, riddles, and the like. If Blue J eventually mastered the game and even defeated Ken Jennings and Brad Rutter in its televised showdown, would its expertise be too specific, or esoteric, for the broader world of business? Would it be flummoxed once it ventured outside its familiar grid of thirty clues? After all, Jeopardy champions were hardly famous for running corporations, mastering global diplomacy, or even managing large research projects. They tended to be everyday people—real estate agents, teachers, software developers, librarians—all with one section of their mind specially adapted—or possibly overfitted—to a TV quiz show.
David Ferrucci spent his days swimming in statistics. They defined every aspect of the Jeopardy project. Blue J’s analysis of data was statistical. Its confidence algorithms and learning programs were fed entirely by statistics. Its choice of words and its game strategy were guided by similar analysis, all statistical. Blue J’s climb up the Jennings Arc was a curve defined by statistics, and when it got into sparring sessions with humans, sometime in 2009, its record would be calculated the same way. The Final Match was the rare exception—a fact that haunted Ferrucci from the very start. Blue J’s fortunes would be defined more by chance than probability. One game, after all, was a minuscule test set, statistically meaningless. A bit of bad luck on a couple of Daily Doubles, and Blue J could lose—even if statistics demonstrated that it usually won.
Ferrucci was constantly analyzing the statistical methodology of teaching and testing the bionic player. One day in the spring of 2008, he came up with a question no one had asked before. Was there any variation, from year to year, in the Jeopardy clues? He asked Eric Brown, the guardian of Blue J’s training set. Did Blue J fare better against the clues from some years than others?
It was odd, looking back, that such a simple question had gone unasked for so long. It could be important. Even a change in the clue writers