A Semantic Analysis Based Automatic Object Classification and Activity Perception System for Large-view Urban Environments

Currently available methods for object recognition and classification primarily rely on static information in single-frame images. However, for the combat aerial video (usually low resolution video), all the static indexes used for object classification and recognition are almost impossible to obtain. To address this challenge, researchers are particularly interested in developing new techniques to determine the type, functionality, purpose or intent of static and moving objects through analysis of dynamic information in persistent surveillance video. The objects of interest include buildings, functional areas, vehicles, and human beings. In this proposal, DCM Research Resources LLC and its collaborator propose to develop an automatic innovative 3D and dynamic semantic scene analysis based approach that exploits surveillance video data mainly captured from UAV platforms to classify buildings, vehicles, and people automatically. In our proposed automatic object detection and classification system, in addition to 3D static object’s visual features (e.g. building’s shape, line orientation, color, and texture) and the 3D static structures of the urban environment, we will also explore dynamic video features which include vehicle/human motion patterns over time. All these static and dynamic features will be used to construct spatial-temporal feature vectors, and the new generated vectors will then be sent to a probabilistic Dynamic Influence Diagram (DID) reasoning model for real-time and automatic building/vehicle classification. Besides the novel algorithms on building and vehicle classification, we also propose a Decentralized Dynamic Markov Game Model (DDMGM) for human activity and intent inference. In addition, we propose to develop novel 3D algorithms on automatic building detection, 3D terrain modeling, and visualization to support accurate object category and classification.