SBIR Phase I: VideoPoints: A Companion for Classroom Learning

This SBIR Phase I project will develop text, speech and image analysis technologies to transform a set of educational lecture videos into an interactive learning companion. A lecture video will be presented as a series of topically cohesive sections, each represented by a textual summary, a visual summary, and a video segment. Students will be able to query a series of lecture videos with the answers presented as a combination of text, images, speech with transcript, and video, along with links to additional relevant resources. These technological enhancements will drive the development of the lecture video management system with capabilities well beyond commercial state of the art. The students will gain the ability to instantly access any information in a semester long course, with little overhead for the instructor. The business plan focuses on a freemium model designed for wide adoption with no cost to instructors and a very low cost to students. The ultimate potential societal benefit is a significantly better learning experience and learning outcomes for higher education students. The key innovations in this project are summarization of video segments and a Questions & Answers (QA) system customized for a series of lecture videos. These are formidable challenges despite existing substantial related research in text mining and image analysis. The information content of a lecture video spans multiple modalities, specifically, screen text, images, and speech. Further, each modality has unique characteristics. Screen text is typically unstructured and includes Optical Character Recognition errors. Transcripts from classroom lecture videos contain informal classroom interactions and suffer from speech recognition errors. The images in a lecture video can represent a variety of concepts including illustrative examples, graphs and charts, and camera images. Innovative approaches to topic modeling will be developed to handle the unique nature of the input compared to most documents. The project will employ representation learning approaches to map from each modality to a common semantic space that will drive the matching between student questions and the content of lecture videos. Since the best answer to a student question may not be fully contained in a lecture video, external resources including textbooks, message boards, and actual quizzes and exams will also be analyzed to drive the QA module. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.