Applying Probabilistic Programming to Evaluate Learning-Enabled Systems (APPELES)

Learning-enabled systems (LESs) that improve their performance over time and adapt to new environments have significant potential for a variety of military applications, such as robotics, intelligence analysis, and cyber defense. To be deployed operationally, LESs must be thoroughly evaluated by test and evaluation (T & E) engineers. In the Applying Probabilistic Programming to Evaluate Learning Enabled Systems (APPELES) framework, we formulate the task of evaluating an LES as a problem of predicting the future performance of the system. We use probabilistic modeling to describe the learning process, the LES, and its environment, to infer characteristics of the LES from data obtained during testing, and to predict its future performance. Our probabilistic models are expressed using probabilistic programming, a powerful framework that enables the expression of probabilistic models with rich data types and control flow. Our models (1) capture a range of learning frameworks, (2) support generalization from limited test runs to longer learning processes, (3) support generalization from testing environments to a broader range of operational environments, and (4) enable the T & E engineer to evaluate the time-varying performance of LESs across measures of interest.