A Scalable and Real-time Hybrid Multimodal Content Exploitation System

For many analysis tasks, the frame-level contextual understanding including query and content tagging on the multimedia data will provide time-critical actionable intelligence and insights. However, such frame-level capability does not exist for multimedia streams and files. To address this need, IAI, along with our collaborators, proposes to develop a Scalable and Real-time Hybrid Multimodal Content Exploitation System: HExp (Hybrid Exploitation). The key innovation is to seamlessly and systematically integrate information retrieval methods in different information spaces through theoretically sound retrieval models so that intelligence analysts can search multimedia contents in both the structured and unstructured spaces. HExp serves as a software system for intelligence analysts to search, tagging and tracking multimedia frames based on the dimensions of time, location, and event categories. HExp consists of three subsystems in different layers: (1) the distributed video/audio annotating and tagging subsystem that transform raw files into text descriptions and semantic entities and relations, (2) distributed hybrid information retrieval subsystem that index the unstructured and structured information from multimedia data using four different retrieval models, and (3) context-sensitive user interaction system that extends IAIs faceted heterogeneous search interface with a persistent user modeling module.