Capturing Experts'Policies to Measure Pilot Performance

ABSTRACT: Real-time pilot performance has proven difficult to measure, due to an inability to capture important interactions between pilot actions and displayed information. Intelligent Automation. Inc."s innovative approach is to capture expert scoring policies by having experts first holistically judge quality of pilot performance. Previous research in other domains has shown that experts"judgments are both reliable and valid. Expert raters then describe the rationale underlying their holistic judgments, making them explicit in discussions with other experts. These rationales identify critical relationships between what pilots see and what they do. For this project, pilots with varying degrees of experience will fly simulated missions using a cockpit simulator while the pilots"eye movements are tracked. Work performance samples will be collected and given to expert pilots to judge. These experts will then discuss underlying reasons for their holistic scores with each other. Experts"reasons will then be classified into categories from existing task analysis, and scoring worksheets which apply expert"s policies to work samples will be constructed and verified. The scoring worksheets will show that pilot evaluations can capture interactions between pilots and displays. The method will be analyzed for its applicability to the F-35 HMD environment. BENEFIT: The direct and immediate benefit of this research is its applicability to real-time Air Force pilot assessment. As part of this SBIR project, the Air Force Research Lab sponsors will see the applicability of this research to their programs, and incorporate it as they see it benefiting their existing training systems. The commercial applicability of this research is to investigate and then publicize the expert policy capturing (EPC) approach. The expert policy capture approach is a systematic, reliable, inexpensive method to bring the expertise collected by experts through years of experience to the assessment of operator performance in complex environments. We have demonstrated this approach in the domain of very complex equipment diagnosis; we are currently applying this approach to the dynamic environment of directing counterinsurgency operations in small villages in the Army"s Contemporary Environment. The current research will apply the expert policy capturing approach to highly perceptual domains that require near-immediate responses to rich displays within complex situations. Thus, this research will investigate the applicability of the expert policy capturing approach to new types of domains. The commercial value of this approach is that as assessment of performance becomes an increasingly desirable training capability and societal goal, methods to inexpensively and authoritatively assess performance in complex environments will be necessary. Applying the expert policy capturing method will be one very efficient method to construct valid and transparent assessments of work sample performance.