Testing AI Learning in Open-world Novelty Scenarios (TALONS)

Artificial intelligence (AI) systems are quickly becoming ubiquitous in both civilian and military products. Self-driving vehicles are a fast-approaching critical application of newly developed AI. Most automotive industry leaders are preparing for a near-term future where autonomous self-driving vehicles are deployed for commercial and personal use. On the military side, self-driving vehicles reduce the risk of loss of human lives and enable new military strategies that were previously impossible. However, existing AI systems in use in the real-world are still unable to contend successfully with novel situations. Multiple ongoing research programs are attempting to resolve this problem by developing AI systems with the ability to respond appropriately, safely, and robustly to novel situations. However, these programs lack scientifically grounded methods to test these applications and more generally, to measure their robustness. The TALONS System conceptualized by Navatek during the Phase I effort provides the theoretical foundation to build a novel scenario generation and evaluation system capable of measuring AI agents’ robustness to novelty in high-fidelity simulated environments. In Phase I, we successfully developed a framework for generating novel scenarios grounded in a new multi-dimensional characterization of novelty. We formally specified this multi-dimensional characterization, metrics for collecting data on AI agents under test, and methods for evaluating agent robustness to different types of novelty. In Phase II we propose to develop the TALONS System based on concepts and prototypes developed in Phase I. We will evaluate both in-house baseline agents and third-party AI agents with respect to novelty robustness. These evaluations will use the CARLA self-driving vehicle simulator (Section 2.2.1), a well-known high-fidelity simulator built using the Unreal Video Game Engine (which among other features offers a high-fidelity physics engine). To support the development of the TALONS System we will extend the scientific theory from the Phase I effort regarding T-transformations, fair evaluation protocols, relevant occurrences of novelty, and minimum required amounts of exposure to novelty to enable learning. Finally, we will develop verification and validation reporting techniques and procedures to support future use of TALONS.