Explainable Artificial Intelligence based Verification & Validation for Increasingly Autonomous Aviation Systems

Artificial Intelligence (AI) algorithms, which are the heart of emerging aviation autonomous systems and autonomy technologies, are generally perceived as black boxes whose decisions are a result of complex rules learned on-the-fly. Unless the decisions are explained in a human understandable form, the human end-users are less likely to accept them, and in the case of aviation applications, certification personnel are less likely to clear systems with increasing levels of autonomy for field operation. Explainable AI (XAI) are AI algorithms whose actions can be easily understood by humans. This SBIR develops EXplained Process and Logic of Artificial INtelligence Decisions (EXPLAIND), which is a prototype tool for verification and validation of AI-based aviation systems. The SBIR develops an innovative technique called Local Interpretable Model-Agnostic Explanation (LIME) for making the learning in AI algorithms more explainable to human users. LIME generates an explanation of an AI algorithm’s decisions by approximating the underlying model in the vicinity of a prediction by an interpretable one. We apply LIME to a NASA-developed aircraft trajectory anomaly detection AI algorithm (MKAD) to provide a proof-of-concept. EXPLAIND represents an important step towards user acceptance and certification of multiple AI based decision support tools (DSTs) and flight-deck capabilities planned to be developed under NASA’s System Wide Safety and ATM-eXploration projects. EXPLAIND also benefits NASA’s planned human-in-the-loop (HITL) simulations of machine learning (ML) algorithms using the SMARTNAS Testbed by providing techniques for making the algorithm’s decisions more understandable to HITL participants. Moreover, with new European Union regulations soon requiring that any decision made by a machine be readily explainable, the EXPLAIND approach is also relevant to multiple non-aviation fields such as medical diagnosis, financial systems, computer law, and autonomous cars.