Multimodal Interfaces for Synthetic Training Environments (MIST)

Current synthetic training environments are limited in their effectiveness by their exclusive use of conventional human-computer interaction techniques (e.g., mouse/keyboard/joystick). To successfully train soldiers in small-unit tactics, trainees need to be able to interact with semi-automated forces through the use of realistic gestural and verbal communication interfaces. To improve training by incorporating realistic interfaces, we propose to design and demonstrate Multimodal Interfaces for Synthetic Training environments (MIST) through the use of a reconfigurable hardware and software system. Four components define our approach: First, we will determine the types of communication needed by analyzing soldier communication needs through studying doctrine documents and interviewing and observing subject matter experts. Second, we will use this analysis to drive the design of multimodal interfaces for use in synthetic training environments. This effort will focus on the use of voice recognition to support verbal commands, and the use of inertial sensors to support gestures. Third, we will design and demonstrate an innovative software and hardware system that will support configuration, selection, and adaptation of multimodal interface methods. Fourth, we will define initial evaluation metrics and develop an evaluation methodology for assessing the utility and efficacy of the proposed multimodal interface methods.