Imaging System and Lipreading Technique for Cockpit Voice Recognition

Voice command will increase the effectiveness of future military aircraft by providing a fast, intuitive, hands-free user interface. For this technology to gain acceptance, systems must recognize speech quickly and accurately in an adverse environment.Human speech perception is strongly influenced by vision. When visual and auditory cues conflict, listeners' perceptions may be different than with either channel alone (the so-called McGurk effect). Researchers have added lipreading information to audio-based automatic recognizers and reported significant improvements in performance. However, lipreading has not been used in commercial recognizers because of two problems: first, it is difficult to interpret the video signal in real-time; and second, certain applications-such as recognition over telephone lines-preclude the use of video.The oxygen mask of the fighter cockpit simplifies the video-interpretation problem. Both an imager and an illuminator-to precisely control lighting-may be incorporated into the mask. Also, because the mask is head-mounted, alignment will be much less of a problem.In this Phase I effort we will incorporate lipreading into a conventional speech recognizer, and we will design and build an experimental mask-mounted imager. In Phase II, we will collect data with a prototype mask, and add lipreading to our own recognizer.