Operations-Responsive Space Objective Nexus using Reinforcement Learning (ORSON-RL)

With missions that include communications, information, surveillance, and reconnaissance (ISR), position, navigation, and timing (PNT), and space domain awareness (SDA), space assets provide indispensable services to US warfighters, forming the backbone of all US operations in all domains. Because of this, the space domain is increasingly contested. To maintain space superiority and freedom of operation in space and thus ensure the success of US missions in space, it is critical that the Space Force ensure the protection of space assets against multiple threats executed by multiple objects. One key aspect to enabling this protection is the ability to process information on mission-relevant timescales and use that information to predict the behavior and consequences of potential threats, especially in the case of multi-threat and multi-target scenarios. One promising avenue for modeling potentially complex behaviors is reinforcement learning (RL), which seeks to train a policy and/or value function for decision making. An RL agent learns much like a living creature, in that it receives a reward, positive or negative, in response to actions it takes and must learn the best actions to take based on that reward. Different types of policies can exist to utilize that reward in different ways, but the most accurate ones favor a policy that allows for both exploration of the space as well as taking the action that is currently producing the best reward. To capitalize on the behavioral prediction power of RL in the context of space battle management command and control (SBMC2), Stellar Science proposes ORSON-RL, the Operations-Responsive Space Objective Nexus using Reinforcement Learning. ORSON-RL will be composed of two primary components: a powerful simulation environment for training that can simulate the observables that are expected to be available and an execution framework that can perform actions within the same environment repeatedly to train the RL agent. For the simulation environment, ORSON-RL will use the government-owned Advanced Framework for Simulation, Integration, and Modeling (AFSIM). AFSIM will provide ORSON-RL with a comprehensive simulation environment encapsulating a variety of multidomain military systems. To provide the execution framework, ORSON-RL will use the government-owned Galaxy simulation software suite. As the number of agents in the simulation increases and behaviors become more complex, the computing power required to train the RL agent increases exponentially. Stellar Science developed the Galaxy framework for massively parallel execution across a variety of computing resources, from desktop computers to petascale supercomputers. ORSON-RL will utilize Galaxy to train RL agents on representative scenarios, thus enabling the construction of a predictive model of multi-agent behaviors. The resulting model will provide warfighters with tactical-speed analytical capabilities to rapidly respond to mission threats in space.