Video Analysis for Nighttime Surveillance and Situational Awareness

Interpretation of video imagery is the quintessential goal of computer vision. The ability to group moving pixels into regions and then associate semantic labels with those regions has long been studied by the vision community. Only recently have thecomponent technologies matured sufficiently to make this goal attainable for well-defined scenarios. We propose a system for semantic interpretation of certain human behaviors in a nighttime parking lot surveillance scenario. The video stream is firstsegmented into moving objects of interest (people, cars) vs. background (ground, sky, buildings, trees, moving foliage, shadows, etc.). Trajectory analysis is then performed on each object, using robust feature tracking and 3D reconstruction. In parallel,body pose analysis is performed by a probabilistic framework that learns mappings from body silhouettes to physical pose. Trajectory and pose information is combined by inferencing over an iconic action grammar. Our approach to specifying interestingactions via behavioral models is novel, as is the identification of unauthorized actions based on top-down inferencing from these models. We will demonstrate using nighttime parking lot imagery from commercial off-the-shelf low-light video equipment. Toincrease robustness of event detection, we will also explore audio information of events such as cars starting. The proposed system and developed techniques are applicable to a wide range of DoD mission and intelligence community areas, including targetrecognition, HumanID, and surveillance & tracking. The technologies developed here could be used to model the behavior of terrorists and identify individuals engaging in illegal acts given a video stream of their motion through the environment. Thistechnology is directly applicable to various current DARPA programs (including Combat Zones that See).