Opportunistic Real-Time Multimodal Sensor Content Exploitation

We propose to integrate and extend multiple capabilities developed under multiple other government-funded R&D programs to create a unique system for multi-modal sensor exploitation, focused on real-time tracking of activities and intent of persons of interest. Our approach will be implemented as a set of plug-ins to an associative database framework for integrated management of large volumes of multi-source data, cross-indexed with analyst and machine-generated annotations and analyses. Key new capabilities will include 1) automation support for object recognition, target tracking, sensor cross-cueing, and 4D scene reconstruction from multiple, possibly overlapping, video sources, supplemented by other INTs (e.g., audio, analyst chat), and 2) tools for collaborative frame-level annotation/tagging and semi-automated summarization of video streams.