Video Retrieval Based on Language and Image Analysis

We propose to develop practical video libraries allowing users to quickly find useful material. Commercial, home, or surveillance video programming captures information in multiple tracks: picture, sound, and closed captioning. Using multiple modalities cataloging information, visual information, and audio information is critical since no one track contains necessary information to identify clips. There is a growing market in extraction of relevant features from separate audio or video tracks of video programming. We will take advantage of these multiple modalities by developing a video library with powerful data fusion capabilities. Both analog videotape and current digital video services provide only linear access to the program material. Interactive search provides non-linear access to the library contents, reducing search time and increasing the efficiency of video library users. We propose to develop search engines allowing complex queries which take advantage of these multiple modalities. Our project will build on exisiting work in digital video libraries which have developed valuable technology for extracting information from individual tracks. While we may need to develop some new feature recognition algorithms for video or audio, we plan to emphasize data fusion to maximize the video library's utility and minimize time-to-market.