Algorithm-Based People Detection and Threat Determination from Passive Infrared and Visible Cameras

In the modern battlefield, an increasing prevalence of light-weight, low-cost electro-optical (EO) and infrared (IR) cameras worn by soldiers, mounted on ground vehicles, and carried by small unmanned aircraft systems (sUAS) extends battlespace monitoring to unprecedented levels. However, simultaneously monitoring all of these video streams leads to information overload for human operators. To autonomously attend to all of these feeds, we propose the Panoptic Guardian VTA (Video Threat Analysis) system. Panoptic Guardian deploys advanced artificial intelligence algorithms to autonomously detect and classify potential threats in real time, while distilling away innocuous distractions. The Panoptic Guardian algorithms are run at the edge, such as onboard a sUAS, and can detect people across a variety of size scales, supports day/night video operations (EO/IR), and can classify a multitude of activities of interest to be disseminated over the network. The system captures a rich set of activities and attributes, building up an ontology of threat indicators so that more-complex threats can be identified, such as operation of anti-tank guided missile (ATGM) systems, coordinated military maneuvers, or burying improvised explosive devices. Its novel decomposition of the threat-analysis pipeline allows more-economical training of deep neural networks without expensive, comprehensive data collections.