HMD-Compatible Mission Performance Measurement System and Tools

ABSTRACT: The Helmet Mounted Display System (HMDS), developed for 5th generation Joint Strike Fighter (JSF) F-35 aircrafts, is a state-of-the-art helmet which virtually displays avionics and sensor data to the pilot through the helmet visor, thus eliminating the need for these data to be displayed separately in the HUD and flat panels in the cockpit. While the JSF HMDS is the most advanced and capable helmet to date, the interaction of the pilot with the helmet has not been thoroughly explored. Our proposed solution is to develop an HMD-Compatible Mission Performance Measurement System that gathers detailed operator data (e.g., interactions and actions; neurophysiological and physiological signals of attention alertness, engagement, stress; communications) and uses context information (e.g., expected operator interactions and actions, the mission, and, if used in the simulation training, the information about the simulation state) to evaluate pilot interactions with HMDS and identify the implications for mission effectiveness. BENEFIT: Our proposed HMD-Compatible Mission Performance Measurement System will allow for real time F-35 fighter pilot performance assessment, as well as for comprehensive after action reviews. In addition, it will benefit training by allowing instructors to identify training gaps and, potentially, to adapt training design to trainee"s individual learning needs. Finally, it will benefit system designers by allowing them to thoroughly evaluate the effectiveness of the system. We anticipate that it will also have commercial viability in a variety of current military and commercial simulators. For example, the Joint Helmet Mounted Cueing System (JHMCS) to be deployed with F/A-18, F-15E, and F-16 aircraft; or the HMD-enabled 3D games and entertainment applications.