Agile Machine Learning in Dynamic Environments for Complex Event Processing - Phase 2

Under this STTR Phase II project, we propose a framework for complex event detection that addresses the limitations of the state-of-the-art by effectively combining deep learning models with neuro-symbolic reasoning. In our approach, we apply a joint deep neural network (DNN) and statistical relational learning (SRL) based framework that combines semantic reasoning and relational learning with data-driven methods for sensor exploitation, trajectory-based feature computation, activity feature extraction, and foundation data to overcome the challenges for complex event detection in dynamic environments. The proposed effort will build upon the technologies developed during Phase I of this project and will leverage expertise of the Novateur Team in the areas of activity modeling and simulation, deep learning-based detection, tracking, classification, and geo-registration of targets in sensor video. The Phase II effort will focus on extending the variety of complex event scenarios, improving the quality of simulated video, incorporation of mobility data and 3D models of a real city to the simulation framework. The Phase II product will be an end-to-end system that can ingest and process streams from multiple sensors and perform complex event detection that span over long time and may not be detected from a single sensor. Test and evaluation will be performed to provide performance metrics and trade-offs of various technology components.