AI Arsenal Management Panel (AIMP)

DoD’s 2018 Artificial Intelligence (AI) Strategy [1] noted that adversaries and strategic competitors such as China and Russia are making significant investments in AI technologies for national security purposes. Since 2018, DoD has made organizational changes and is investing billions of dollars to incorporate AI technology into its operations. However, there exists a potential for unintended consequences and risk of mission failure when AI is being implemented without technical oversight or accountability measures as part of larger decision-making systems including the DoD Joint All-Domain Command and Control (JADC2) and the Department of Air Force Advanced Battle Management System (ABMS). Any AI technology considered to be used in high-stake settings must undergo rigorous evaluation of its underlying technology and domain relevance in order to quantify its potential risks and utilities. YRIKKA has created a proprietary scoring engine that automates the evaluation of AI technologies on a multitude of criteria including performance, interpretability, and adversarial robustness. Through an application programming interface (API), the engine collects relevant details of the AI product from its developers and domain-relevant information from domain experts. The scoring engine then generates an AI Readiness (AIR) Score for the product and provides reports on the standing of the product with respect to each criterion. This Score provides an all-encompassing overview of the readiness and risks of an AI technology for real-world deployment together with feedback on how to improve the technology. The utility of the AIR Score for AI decision-making is similar to that of the FICO Score used in consumer finance. To empower defense users with an interest in the adoption of AI, YRIKKA seeks to create an AI Arsenal Management Panel (AIMP). AIMP will be populated with AI products that are evaluated through the scoring engine and received an AIR Score. AIMP will be equipped with a search engine that can help users find relevant AI solutions to address their needs. The user can then easily compare potential solutions as well as perform regular audits of their mission readiness using the respective products’ AIR Scores. AIMP and the scoring engine together can enable DoD to adopt trustworthy and reliable AI at scale. In this Phase I proposal, we seek to conduct customer discovery among DAF end-users, including but not limited to the Air Combat Command and the Air Force Research Laboratory, in order to identify existing challenges and pain-points within the current AI Product and Acquisition Lifecycle. In doing so, we aim to incorporate end-user feedback on the viability of using AIMP as a central repository dashboard for auditing and assessing AI technologies before and after their integration into mission-critical decision-making systems. [1] DoD, Summary of the 2018 DoD AI Strategy: Harnessing AI to Advance Our Security and Prosperity (2018).