AI-Enabled Test & Evaluation Toolkit for Pilot Multi-Function Displays

As adversaries to the United States improve their combat aircraft and air defense systems, the Department of the Air Force (DAF) must continuously improve its fleet. An aircraft, when it enters into service, can operate for decades before being decommissioned. During this lifespan, the avionics, flight computers, and other systems receive upgrades to keep aircraft flight ready and mission capable. Software changes, even innocuous updates, take on new import in military systems, where the implications of poorly executed code can have catastrophic results for the pilot, crew, and mission. To mitigate these risks, the 309th Software Engineering Group (309th SWEG), which is responsible for developing “cradle-to-grave” software and hardware support systems, conducts thorough testing before pushing new software to operating aircraft. This process includes recording and reviewing hundreds of hours of video from inside a cockpit testbed to observe the software in operation in a relevant environment. This entire process can add months and sometimes years to mission critical upgrades for aircraft, such as the A-10, F-16, F-22, F-35. Working with 309th SWEG, CrowdAI proposes to leverage human-machine teaming to reduce not only the analytic burden on Airmen to perform this task, but to reduce the time to completion while delivering as good or better results. Computer vision (CV) is a field of artificial intelligence (AI) that enables computers to derive meaningful information from digital scenes, such as video— and to take actions or make recommendations based on that information. The proposed project would implement a CV solution to assist Airmen reviewing video of the cockpit testbed during software testing. CrowdAI is an industry leader in an advanced CV technique called image segmentation, which analyzes video at the pixel level. This capability is critical given the complexity of the “heads-up display”, which conveys many pieces of information, projected onto a dynamic background. CrowdAI was the first company to apply image segmentation to video, earning it the top place in the DAF’s HyperSpace Challenge in 2018 as well as selection by the DOD’s Joint Artificial Intelligence Center as its first CV vendor.  For this proposed work, CrowdAI would develop new CV models for use by the 309th SWEG. The project would include gathering relevant video data, labeling relevant objects in that data for the model to learn, and training multiple computer vision models. These models will then be transferred from the development environment to an operations environment not only to explore cross-domain approaches and technical requirements, but to evaluate model performance on production data. AFRL will provide compute resources during this phase. 309th SWEG is the project end-user and transition partner for integrating into the Advanced Battle Management System (Party Bus) for anticipated long-term sustainment and which CrowdAI is already on contract (FA8612-21-D-0117).