Sequential Phase II to develop techniques and technologies to increase AI data labeling efficiency and accuracy

In our original effort under this SBIR Phase II, EI focused primarily on measuring the efficiency and effectiveness of various annotation geometries and quality assurance processes.  For example, we set up experiments that compared the labor time and annotation accuracy of creating computer vision (object detection) training data from satellite imagery using bounding box annotations vs. polygon annotations.  We then compared the performance of the AI models developed using each type of training data.  We also performed experiments testing different quality assurance (QA) methods.  These QA procedures are designed to reduce annotation and classification errors in test and training data sets.  The goal of the extended program is to provide NGA with deeper understanding of time/resource/computational costs/and performance tradeoffs of various labeling methods, processes, and tools and their impact on ML training efficiency and AI model performance. These insights will help NGA implement processes to significantly reduce the time from sensor collect to deployed AI model.  NGA would also be able to use these analyses to develop and implement future internal data curation, labeling strategies, AI model development workflows, and AI testing plans to more quickly build more reliable AI technologies to support various analyses and intelligence products. For the extended program, EI will employ process optimization techniques and analyses to accurately and precisely measure the time and resources needed to create effective training and test data sets.  We would start with mapping the current sensor data collection - training - data creation - model development and testing -model deployment process and identify key bottlenecks or areas that take the most time to complete.  We will focus primarily on the early phases of this process: data acquisition, curation, cleaning, and labeling.  We will also investigate the AI model training / iteration phases and ways in which we can shorten the time needed to iterate model training.  Following TQM practices, EI will work with the NGA team, to complete Pareto type analyses to identify the parts of the process that take the most time and/or resources and focus our attention on these initially. EI will use our has existing annotation and model development pipelines and workflows as they already collect precise metrics that will enable these process analysis efforts, including: Data transfer, downloading, chipping, and uploading time Annotator training development and delivery time Annotation time by data type, ontology, geometry, and QA process Error rates by annotator, data type, annotation, and classification Measure data/label quantities needed and/or used for model training and testing