Holistic Extensible Real-time Multi-modal Integrated Tracking (HERMIT)

Current tracking software and algorithms are typically limited to forming tracks from individual sensors. When multiple sensors are present, data fusion and track correlation can improve tracking performance, but holistic incorporation of multiple distinct sources of information (OSINT, text reports, patterns of life, doctrine, etc.) is not possible in standard tracking models. This work is intended to further improve computer-aided tracking algorithms by seamlessly incorporating data from multiple sensors and widely varying phenomenologies, contextual clues, and available sensor meta-information. The goals of this effort are to develop a framework and algorithms that can infer target tracking capabilities over time, incorporate known target-track doctrine, and infer novel target track doctrine while learning from extremely limited sets of labeled data from disparate sensors over large geographical regions. We propose two new technologies: text-to-kinematics models for inferring target tracks from textual descriptions, and a universal entity descriptor for cross-modal data sharing. To account for widely varying uncertainty between sensors, our approaches explicitly estimate posterior densities on estimated track and descriptor vectors, enabling principled statistical model updates. Our approach will enable integration of all available sensing data into a holistic tracking framework.