Automated Scene Understanding

Automatic extraction and representation of visual concepts and semantic information in scene is a desired capability in any security and surveillance operations. In the proposed effort we seek to advance the foundations of representation and fusion of data, information and knowledge gathered from diverse sources, and across multiple application domains. We target the problem of visual event recognition in network information environment, where faulty sensors, lack of effective visual processing tools and incomplete domain knowledge frequently cause uncertainty in the data set and consequently, in the visual primitives extracted from it. We adopt Markov Logic Network (MLN), that combines probabilistic graphical models and first order logic, to address the task of reasoning under uncertainty. MLN is a knowledge representation language that combines domain knowledge, visual concepts and experience to infer simple and complex real-world events. MLN generalizes over the existing state-of-the-art probabilistic models, including hidden Markov models, Bayesian networks, and stochastic grammars. Moreover, the framework can be made scalable to support variety of entities, activities and interactions that are typically observed in the real world. Experiments with real-world data and domain knowledge, in a variety of marine and urban settings, illustrate the mathematical soundness and wide-ranging applicability of our approach.