Valor: A Platform- and Model- Agnostic Evaluation Store

To strengthen the Army’s objective confidence in AI/ML model selection and monitoring, Striveworks proposes a direct to Phase II SBIR to accelerate the expansion of Valor, our industry-leading open-source testing and evaluation (T&E) framework. Valor leverages rich metadata (e.g., time, geolocation, sensor configuration) to filter and fine-slice model performance, informing model comparisons and uncovering potential model vulnerabilities before our adversaries find them. Ultimately, this T&E capability maturation will produce tooling to provide more timely and relevant intelligence to support tactical and strategic decision-making.   Valor is fully integrated into Striveworks’ Chariot machine learning operations (MLOps) platform, which is Army-fielded and battle tested. The Phase II SBIR confidently builds upon these already scalable capabilities, mitigating development and deployment risks.  Striveworks will optimize Valor for speed during evaluation of various AI/MLmodels (CV, LLMs/RAG) and integrate it with our Chariot MLOps platform to improve usability and enhance model selection and monitoring. Integrations with Chariot include: no-code UI support for object detection, semantic segmentation, instance segmentation, and classification T&E workflows; live dashboards for model comparisons to make model evaluation more approachable to non-technical stakeholders; and remediation pipelines that use automated T&E to detect model failure, trigger retraining pipelines, and redeploy the updated model.  Striveworks will advance Valor integrations, including dataset/model holdings and complementary tooling (e.g., adversarial robustness techniques, drift detection, and DevSecOps pipelines). Striveworks will provide hands-on access to Linchpin through an integrated Striveworks Chariot license direct to Linchpin for the duration of the Phase II SBIR PoP, ensuring responsive feedback and assessment of Valor’s effectiveness and the useability of this needed T&E framework. Our technical approach prioritizes speed, efficiency, ease of use, scalability, overall robustness, and seamless integration with DOD systems. To meet the Army’s needs for safe, secure, and trustworthy AI/ML, we will optimize Valor’s model evaluation, extend its no-code user interface (UI), incorporate automated T&E into model remediation, and integrate Valor into Army systems. As a result, Valor will serve as a seamless component in an end-to-end AI/ML stack designed to enhance model development and greatly improve post-production maintenance, including remediation, for both defense and commercial operations at scale.