VISCARO - Video Information Summarization, Captioning, Analysis, and Rank Ordering

Military and open-source videos contain a wealth of information about entities, concepts, and events of interest to US and coalition forces. Intelligence analysts have access to these video data sources but are severely limited in exploiting the information contained within due to manpower and technical limitations. Unlike most data sources, video data is actually multi-modal. Relevant information can be found in the video stream (individual image frames), the audio stream, text that is embedded into the image frames, and metadata text that may accompany the video. Most significantly, none of these individual modes typically describes the context and meaning of the video completely. It is the combination of all of them, fused together, that fully conveys the true story provided by the video. To address this challenge, the DECISIVE ANALYTICS Corporation (DAC) team proposes to develop an automated, multi-modal video summarization tool called the Video Information Summarization, Captioning, Analysis, and Rank Ordering (VISCARO) system. VISCARO will fuse data from all modalities into a common semantic data model to enable automated generation of text-based summaries. Additionally, VISCARO will provide a natural language search capability that will search a corpus of videos and return rank ordered videos pertinent to their query.