Intelligence agencies have a limited number of trained human analysts looking for undeclared nuclear facilities, or secret military sites, hidden among terabytes of satellite images. But the same sort of deep learning artificial intelligence that enables Google and Facebook to automatically filter images of human faces and cats could also prove invaluable in the world of spy versus spy. An early example: US researchers have trained deep learning algorithms to identify Chinese surface-to-air missile sites—hundreds of times faster than their human counterparts.
The deep learning algorithms proved capable of helping people with no prior imagery analysis experience find surface-to-air missile sites scattered across nearly 90,000 square kilometers of southeastern China. Such AI based on neural networks—layers of artificial neuron capable of filtering and learning from huge amounts of data—matched the overall 90 percent accuracy of expert human imagery analysts in locating the missile sites. Perhaps even more impressively, the deep learning software helped humans reduce the time needed to eyeball potential missile sites from 60 hours to just 42 minutes.
"The algorithms were used to find the locations where they said there is a high confidence of a missile site, and then humans reviewed the results for accuracy and figured out how much time the algorithms saved," says Curt Davis, a professor of electrical engineering and computer science, and director of the Center for Geospatial Intelligence, at the University of Missouri. "To my knowledge that’s never been studied before: How much time did you save, and how does that ultimately impact the human performance?"
The University of Missouri study, published on October 6 in the Journal of Applied Remote Sensing, comes at a time when satellite imagery analysts are figuratively drowning in a deluge of big data. DigitalGlobe, a leading commercial satellite imagery company, generates about 70 terabytes of raw satellite imagery each day, never mind all the imagery data coming from other commercial satellites and government spy satellites.
Davis and his colleagues showed how off-the-shelf deep learning models—heavily trained and modified for satellite imagery analysis—could identify objects of potentially great interest to intelligence agencies and national security experts. The deep learning models, including GoogleNet and Microsoft Research's ResNet, were initially created to detect and classify objects in traditional photo and video imagery. Davis and his colleagues adapted such models to the challenges and limitations of interpreting satellite imagery, such as training some deep learning models to interpret both color and black-and-white imagery, in case only black-and-white images of SAM sites were available.
They did so with satellite imagery representing a huge swath of Chinese territory, not that much smaller than the entire country of North Korea.
And in fact, analysts rely extensively on satellite imagery to keep track of how North Korea's weapons programs evolve. Human analysts have already likely identified most, if not all, existing SAM sites within the relatively small country. But similar deep learning tools could help automatically flag new SAM sites that appear in North Korea or other countries. Knowing the location of existing and new SAM sites can sometimes lead analysts to other locations of interest, because countries often place SAM sites in specific areas to defend valuable nearby assets from air attack.
The latest study also illustrates the challenges of applying deep learning AI to satellite imagery analysis. One major problem is the relative lack of large training datasets that include the hand-labeled examples needed to train deep learning algorithms to accurately identify features in satellite imagery. The University of Missouri team combined public data on the worldwide locations of about 2,200 SAM sites with DigitalGlobe satellite imagery to create their training data, and then tested four deep learning models to find the best-performing one.
The researchers ended up with only about 90 positively identified Chinese SAM site examples to train their AI. Such a puny training dataset might normally fail to yield accurate deep learning results. To get around that problem, Davis and his colleagues transformed the 90-odd training samples into about 893,000 training samples by shifting the original images slightly in different directions.
The impressive deep learning performance in the study likely benefited from SAM sites being fairly large, and having distinctive patterns when viewed from above in satellite images. Davis cautioned that deep learning algorithms face a much greater challenge when trying to analyze smaller objects such as mobile missile launchers, radar antennas, mobile radar systems, and military vehicles, because the available satellite imagery data will have fewer pixels to work with in extracting identifying features.
"It is an open question in our mind how well convolutional neural networks will work on smaller scale objects like this, especially when tested against large area datasets like we did with the China study," Davis says.
Even imperfect AI tools could prove incredibly helpful for intelligence gathering. For example, the International Atomic Energy Agency has the unenviable task of monitoring all declared nuclear facilities and also searching for undeclared facilities among nearly 200 countries. Deep learning tools could help the IAEA and other independent organizations use satellite imagery to monitor development of nuclear power and related weapons of mass destruction, says Melissa Hanham, a senior research associate in the East Asia Nonproliferation Program at the Middlebury Institute of International Studies at Monterey, Calif.
"We're in a world where there is just so much data that the best way to approach it is to do a good job on a lot of it rather than a perfect job on a small bit of it," Hanham says. "I'm looking forward to automating all the tedious and redundant parts of my job."
