Machine Readable Contextual Understanding and Drilldown

Under this SBIR Phase I project, we propose a framework for complex event and anomaly detection that addresses the limitations of the state-of-the-art by effectively combining deep learning models with large language models. In our approach, we combine the power of supervised and unsupervised DNNs to discover hidden features from individual data streams with the ability of large language models to perform reasoning in a noisy, multi-variate, correlated, and constrained setting while incorporating a corpus of contextual cues and domain knowledge. The Phase I effort will focus on developing a simulation framework to generate realistic scenarios using our existing large datasets of surveillance and webcam videos as well as AIS and GPS tracks that span over several years. In addition, we will incorporate multi-intelligence (multi-INT) and Open-Source intelligence (OSINT) data sources over extended periods of time with foundation GIS datasets. These scenarios will serve as test dataset for the proposed complex event detection system, developing an end-to-end system that can ingest and process streams from multiple sensors and perform complex event detection. The proposed effort will build upon the experience and expertise of Novateur team in the areas of deep neural networks, contextual modeling, data exploitation, and activity and behavior modeling.