Automated Scene Understanding: Recognizing Function from Form and Motion

Recognition of objects and their relationships is a critical part of automatic scene understanding. Scene understanding based on object appearance alone is not sufficient as the functions of a facility change over time, especially in areas of ongoing conflicts. However, almost all surveillance imagery contains vehicular and human activities. By monitoring the activities of vehicles and dismounts, we can better recognize the true functions of buildings, vehicles, and facilities, and hence understand the scene more accurately. SET Corporation proposes to develop a scene monitoring and event recognition toolbox for activity and facility recognition through functional interpretation. We will start with detection and tracking of all moving vehicles and humans, recognize the activities that humans and vehicles are involved with, and finally determine the nature of the facilities with which humans are interacting. If successful this novel approach will constitute a fundamental advance in video understanding, with applications in reconnaissance and surveillance for counter-terrorism and counter-insurgent operations.