Spatiotemporal Deep Learning for Full Motion Video Exploitation with On-the-Move EO/IR Sensors

PhysicsAI proposes to implement, evaluate, and demonstrate a spatiotemporal video perception capability based on deep neural networks (DNNs) for exploiting full-motion EO/IR sensors mounted on vehicles to provide a real-time on-the-move threat detection capability. The Phase II prototype builds on the spatiotemporal exploitation pipeline architecture developed in Phase I including: (1) End-to-end deep learning – for mapping EO/IR sensor inputs directly into outputs resulting in more accurate, faster, and more robust detection, (2) Spatiotemporal deep learning – for providing a fully-integrated spatial and temporal machine perception of robust target features that evolve over time in full-motion EO/IR sensor data, (3) Multi-branch neural networks – for fusing data from multiple EO/IR sensors in real-time and for producing multiple types of outputs such as object classification, location, and tracking. In Phase II, PhysicsAI will develop a prototype mounted on ground and aerial vehicles to demonstrate real-time on-the-move threat detection in representative mission scenarios. PhysicsAI also will develop a software toolkit that uses a combination of supervised, unsupervised, and transfer learning techniques to train spatiotemporal DNNs using a combination of labeled/unlabeled real-world data and synthetic data. We anticipate the Phase II effort will increase the TRL from 3 to 6.