Total Recall - C. Gordon Bell [33]
“Anything unusual today?” inquires the intelligence officer.
“Pretty calm except that the children were acting strangely,” he replies.
“How do you mean?”
“It’s kinda hard to describe. . . .”
In Iraq, today’s soldiers are fighting an insurgency that uses civilians for cover. According to the soldiers we interviewed, the most common point of contact with the enemy is the Improvised Explosive Devices (IEDs) used against vehicles and troops. Soldiers are forced to uncover the enemy through everyday patrols and intelligence gathering. However, the soldiers are facing an information shortage; they are not equipped to gather this type of everyday intelligence. Soldiers also need a means to share information with intelligence officers and between patrols. Currently, this information is mostly conveyed orally or through images taken with the soldiers’ personal digital cameras. Georgia Tech’s Soldier Assist System (SAS) attempts to augment this process by automatically capturing a “blog” of a soldier’s patrol and allowing him to rapidly select media from that patrol to share with his intelligence officer.
To understand the goal of SAS, let’s revisit the above scenario. While on patrol, the patrol leader and each of his two squad leaders wear SAS capture hardware. Each system records high-resolution images from a head-mounted camera, two streams of audio (one from a close-talking microphone and one from a chest-mounted microphone to record ambient sound), location using the Global Positioning System, and the soldier’s movement using accelerometers on the wrists, hip, thigh, chest, and weapon. During the patrol, the soldiers can also use their high-resolution manual camera to capture images they feel may be important later. Upon returning from patrol the platoon leader now has the information to answer the intelligence officer’s questions:
“Pretty calm except that the children were acting strangely.”
“How do you mean?”
“It’s kinda hard to describe, but let me show you.”
The platoon leader now looks at a map with his GPS path overlain. He selects the area around a local mosque where he met the children and scans for an appropriate image. Looking for images where the system indicates he was speaking, he quickly finds an image with the children’s faces to the intelligence officer.
“You see, generally the children will come up to us along the road because they know we carry candy for them. But today they are here along the wall of the mosque,” states the platoon leader.
As the debriefing continues, the intelligence officer sees a suspicious white pickup truck in the background of one of the platoon leader’s images. While the platoon leader’s blog does not have a good image of the truck and its environment, he uses SAS’s automatic annotation system to select images from his squad leaders’ blog where they “took a knee” to provide security while he was talking with the children. (Often, when monitoring the environment to provide security, the soldiers support themselves on one knee while maintaining a good field of view.) The platoon leader quickly finds a good image of the truck and shows it to the intelligence officer.
“I bet you the owner of that pickup truck was there just before you and was scouting the area for the insurgency. Let’s record this license plate and give it to the next patrol to look for,” says the intelligence officer, ending the debriefing.
The Georgia Tech team found that, for the soldiers, “there was no such thing as ‘too much information’ for presence patrols.” However, with all the data their system could collect, they didn’t want the soldiers spending many extra hours wading through it all to find relevant parts to report. So, they did some postprocessing to automatically detect and tag activities like raising a weapon, walking, running, crawling, standing, shaking hands, driving, opening a door, and so on. These are not intended to replace the intelligence, intuition, memory, and common sense of soldiers, but to complement and enhance them. By combining