DR. LEGS: Deep Reinforcement Learning Ecosystem for Guaranteed Safety

To bring Reinforcement Learning (RL) a step closer to adoption in spacecraft control systems, Rhea Space Activity, Inc. (RSA), the Assured, Reliable, and Interactive Autonomous (ARIA) Systems Group, and the Autonomous Vehicle Systems (AVS) Lab at the University of Colorado, Boulder propose DR. LEGS: Deep Reinforcement Learning Ecosystem for Guaranteed Safety. DR. LEGS addresses the fundamental issue of RL for safety-critical applications—lack of performance guarantees and mature verification methods—by building safety requirements directly into the Neural Network (NN) training process. To do so, DR. LEGS leverages two correct-by-design methodologies with which ARIA Systems Group has deep knowledge and experience: adversarial training with barrier functions and neuro-symbolic methods. DR. LEGS’s Phase I effort will produce a comprehensive set of tools and theorems for RL control with performance guarantees, and will also add Deep Reinforcement Learning (DRL) and logical requirements capability to Basilisk (an astrodynamics and space vehicle simulation framework developed by the AVS Lab), allowing for immediate spacecraft simulation development in a Phase II effort. With the DR. LEGS framework in place, spacecraft GNC engineers may leverage RL agents and their versatile representational ability of nonlinear maps and fast computation for sophisticated guidance and control problems.  Compared to conventional, ad-hoc approaches to safety-critical RL such as guards or shields, DR. LEGS and its proposing team present the following advantages:Guarantees Built into Model Training: Adversarial training with barrier functions simultaneously trains the controller NN and the barrier NN to achieve verification-in-the-loop synthesis, while neuro-symbolic methods use formal languages to specify safety requirements as logical formulas that are directly incorporated into the training framework.Immediate Extension to Spacecraft Simulation Development: The AVS Lab has developed Basilisk, a Python-based, modular spacecraft simulation framework with numerical simulation and reinforcement learning capabilities, which will be used for implementing and testing DR. LEGS’s techniques. Collaboration with AVS Lab allows for direct access to experienced Basilisk developers and capability enhancement for safe-by-design DRL.Operationally Relevant RPOD Expertise: RSA brings expertise developing RPOD-specific safety analyses and operating human-rated RPOD missions. This includes comprehensive understanding of safe guidance and control design in conjunction with extensive operational experience and the insight required to work towards intelligently designed, minimal RL structures rather than incomprehensibly large “black box” design structures.