Mission Assurance for Autonomy and Reinforcement Learning with Carnegie Mellon University

Reinforcement learning (RL) is essential for future autonomous flight systems such as those contemplated by Replicator, Shield AI, and others. Unfortunately, RL is incompatible with safety and airworthiness standards like MIL-STD-882E and DO-178C. In partnership with Carnegie Mellon, we wil bridge this gap with a novel assurance case, massive statistical evidence, calibrated metrics of trust, as well as existing level-of-rigor tasks from safety engineering. We already provide autonomy safety assessment to the U.S. Army’s XM30 and RCV programs, as well commercial developers Aurora, BMW, and Mobileye. We wrote UL 4600, the world’s first standard for autonomy safety. We inform T&E policymaking at the DoD and state- and federal-level departments of transportation. We recently completed a study of metrics for autonomy trustworthiness for the Center for Calibrated Trust Measurement and Evaluation (CaTE) run by Dr. Kim Sablon, the Principal Director for Trusted AI and Autonomy in OUSD(R&E). Our objectives on this project are below. We will document research results for the customer, publish academically, and enhance Edge Case’s nLoop SaaS platform, which we make available to DAF end-users on Azure through our partnership with Microsoft. AI Alignment. Trustworthy RL learns policies that satisfy mission-critical requirements. Primal-dual style approaches suffer from instability issues and lack optimality guarantees. We will apply lessons learned from research on inverse constraint learning, inverse reinforcement learning, and moment-matching frameworks. We will expand our research on an E-M approach for incorporating constraints naturally and decomposing the larger RL problem into a convex optimization phase and a supervised learning phase, for stable training performance. Hazard Assessment. This involves evaluating the safety guarantees of policies generated by RL to identify conditions in which an autonomous system might fail to uphold trust. Encountering all mission-critical scenarios is infeasible, so we seek edge cases using distributionally robust optimization techniques. We will iteratively change the starting state policy to find a potentially unsafe problem instance using multiplicative weights methods. Continuous and high-dimensional spaces can be reduced to simpler optimization problems (e.g., supervised learning) through substructure identification. Assurance and Metrics. We will trace evidence from the above work to claims in our Assurance Framework for Trustworthy Autonomy produced for CaTE. We will employ an eliminative argumentation process to improve our assurance case with feedback from end users, subject matter experts, and stakeholders from DAF and DoD programs of record. We will also incorporate templated metrics of trust from the CaTE assurance framework into the AI Alignment activities, making RL training more successful and efficient in producing models that are worthy of our trust.