A Media Integrated Information Retrieval System for Video

This proposal aims to investigate methods for development of a video browser using television news as an application domain. A prototype software will be built based on the existing Virage software framework for content-based image retrieval. In this project we will: 1) Create a set of fully automated video indexes based on in-house and state-of-the-art techniques of shot, motion, and audio segmentation, key frame extraction and analysis. 2) Extract a set of key expressions from the transcribed text of the news, displayed captions and interactively or manually entered annotations where necessary to index the input video stream. We will investigate the extent to which video segmentation techniques can be improved by using key expressions extracted from news transcripts. 3) Develop a visual thesaurus structure to associate related key expressions, related image features and key expressions with image features. 4) Develop a method to use the visual thesaurus to jointly index images and word expressions. 5) Develop an abstraction scheme to relate various degrees of abstractions of the news video (e.g.: headlines, facts, and details). 6) Develop a query processing and browsing schemes to retrieve information at a user-specified level of abstraction using linguistic expressions, image similarity, sequence similarity, and their combination.