Automated Scene Understanding

ObjectVideo, Inc. proposes to develop video inference algorithms and a modular architecture for scene understanding based on the computational framework of Spatio-temporal And-Or Graph (ST-AoG). Various image inference modules can be integrated to the framework through an API for scene understanding. The system can be easily upgradeable as new image processing modules are developed. The architecture also supports the input of user-provided context to improve inference. Leveraging earlier work on semantic annotation, we will develop algorithms to infer complex relationships between scene entities. Plain text reports of the scene are automatically generated to describe these relationships, contextual information, and events of interest. To achieve a high compression rate for bandwidth-limited applications, the text description can be used to synthesize image and video to provide a representative rendering of the scene and events. In addition, we propose a method for indexing hierarchical data as well as a scalable framework for searching a large imagery dataset.