Human Automation Teaming for Efficient Knowledge Extraction and Test Generation

Effective tools for evaluating the proficiency of warfighters at employing complex systems in an operational setting are essential to ensure operational capability within the Navy and other branches of the DoD. Generating exams with answer keys is a time-consuming process, made more difficult by the complexity of both the systems and the operational employment of those systems.  The burden is further increased when systems are updated frequently.  The proposed AI-enabled Supervised Assessment Generation (AI-SAGE) software tool will provide a general-purpose capability to efficiently generate test material from technical documents that describe the capabilities of a complex system and the operational employment of that system.  It will enable a person with domain knowledge related to a particular system (but no AI expertise) to effectively team with AI to complete the test generation task vastly more efficiently than would be possible manually, while achieving accuracy and reliability that would be impossible in a fully autonomous system.  The tool will support both updates of existing test questions and answers based on updated source documents, and generation of novel questions specifically targeted at assessing knowledge of new and updated system features and operational procedures. The Phase I research effort successfully demonstrated the feasibility of the proposed tool by demonstrating key underlying capabilities including question generation, question answering, and question updating.  Phase II work will fully implement the AI-SAGE software tool, and add capabilities including answering/updating/generation of advanced question types and questions over graphics (e.g., diagrams, plots).  It will also include extensive testing to validate the quality of tool outputs as well as the usability of the tool.