Saliency Annotation of Image and Video Data

ABSTRACT: With the current flood of surveillance data available to ISR analysts, human attention has become the most valuable resource to ISR systems. Although automated tracking and labeling algorithms are now capable of automatically identifying and roughly classifying targets, the current rate of false alarms and irrelevant annotations makes existing technology unsuitable for wide-area persistent surveillance applications, where analysts are overwhelmed by irrelevant data. What is needed is a system that incorporates a user-trainable relevance/saliency classification algorithm with the best available tracking algorithms to achieve very low clutter rates even in urban environments. Toyon Research Corporation and Penn State Professors David Miller and George Kesidis propose to address this need through a prototype system for automated saliency annotation, incorporating recent results in active learning of semisupervised mixture models, and automated feature extraction from video data with reconstructed 3D models. The proposed system combines the extremely low clutter rates of Toyon"s 3D clutter suppression algorithm, with high-accuracy classification methods using fine-grained mixture models developed by Professors Miller and Kesidis. BENEFIT: The capability generated by the proposed system will be crucial to Air Force ISR systems that rely on real-time processing and mining of wide-area persistent surveillance (WAPS) data, by dramatically increasing the effective surveillance region size per operator. The technology also reduces potential tactical cost in the consequences associated with misidentifying critical targets (either missing terrorist activity, or incorrectly targeting innocent civilians). Active learning has the potential both to greatly reduce the amount of labeling analysts need to do to achieve accurate automated classification/saliency determinations and, by achieving accurate classification, to reduce the risk of adverse tactical consequences. Additionally, the technology has many non-military applications, including manufacturing, construction, security, and automobile traffic monitoring, where the rarity of salient events limits the effectiveness of existing wide-area video systems from being effective, due to shortage of human resources.