Validation and Evaluation with Incomplete Labels (VEIL)

Deploying trustworthy AI/ML models at scale requires frequent and accurate assessment of model performance to ensure that they meet mission needs. Yet, assessment of model performance ordinarily requires exhaustively labeled test data that is free of errors. ôPerfectö test data is both completely and accurately labeled û two qualities that are idealistic to assume and expensive to ensure. Organizations which manage sizable portfolios of AI/ML models must know the expected performance of their deployed models without prohibitive expenditures of time and labor to create perfect test data.áEtegent Technologies and CACI International propose VEIL (Validation and Evaluation with Incomplete Labels),áa modular, modality- and model-agnostic two-phase software stack to enable test and evaluation with imperfect test data. In this system, the first layer is label-masking software (Corrective Lens for Testing, COLT) which suggests truth labels and predictions to ignore. The second layer is a model evaluation toolkit (Algorithm Evaluation Suite, ARES) which produces noise-resistant metrics for geospatial object detection. This solution reduces the generalization gap between metrics produced on perfect and imperfect data for any object detection model in any modality. Further, its modular structure (COLT and ARES can be used independently) enables incremental adoption and eases integration with existing systems.áEtegent Technologies has been providing geospatial computer vision solutions for the DoD/IC for the last decade. Its ARES tooling, for instance, is used for training and evaluating deployed SAR and EO object detection models at NASIC for the last 4 years. Furthermore, the Etegent teamÆs relationships with NGA Rare Objects, proximity to the SAFFIRE program, and day-one access to TS-level facilities and workspaces position them well for immediate production and future transition of the proposed system.ááCACI brings extensive experience and resources to this program, including customized processes and architectures for AI/ML application development and operations. Their significant investment in internal research and development since FY 2021 has positioned them at the forefront of AI innovation. Their many years of experience with NGA Rare Objects and NGA programs of record puts them in a unique position to bridge research programs and provide cross-program benefits through this effort.ááReliable and cost-effective testing and evaluation with imperfect test data is a challenging problem facing the DoD/IC that has been underserved by academia. Etegent and CACIÆs proposed two-phase solution to produce noise-resistant metrics will give users the information they need to inform model deployments without the time and expense of chasing perfection.