Scalable Systems for Training and Readiness using Artificial Intelligence and Computer Vision

Advances in AI and computer vision are creating new opportunities to transform the training and supervision of military personnel performing tasks that underlie operational readiness, ranging from medical testing to aircraft and ship maintenance. This STTR proposal between Saddlepoint Labs LLC and Cornell University will initiate the development of a general-purpose plug-and-play algorithmic framework that automatically captures the interactions of objects and actions as an expert performs a task, and then rapidly scales dissemination of that captured expertise as training for a complex organizational work force. The successful execution of the STTR Phase I objectives has the potential initiate a transformation in the way that DoD approaches training and supervision of tasks across a large and geographically distributed workforce, advancing agility, operational readiness and resilience of the warfighter and other military personnel.  The algorithmic framework for task training and supervision to be developed in this STTR will provide rapid generalization to new tasks, similar to a foundation model, with key characteristics of a custom-built computer vision pipeline: interpretability, accuracy, real-time inference on wearable hardware, and modest training-data requirements.  The core of the framework will be algorithmic building blocks that can be rapidly created and then assembled to provide different kinds of functionality (Objective 1). These building blocks will operate on frames from a video sequence to extract key information in a form optimal for supervision of manual tasks. Other building blocks will combine this information with input captured from an expert on common errors and appropriate feedback to generate guidance, and an auditable record of how the task was performed (Objective 2). Overall, the framework will function like a programming language for a task-training and supervision system. We will test and validate this framework in the Phase I Based Period in the important context of medical diagnostic testing widely used by DoD (Objective 3).  Specifically, we will address error-prone pipetting of small volumes (microliter) of liquid reagents and samples to/from microwell plates, as routinely done for PCR and ELISA assays. Our system will identify errors and give feedback to personnel performing the work.  During the Phase I Option Period, we will further develop and test the generality of our approach to include transfer of fluid to and from Eppendorf tubes, beakers and bottles; and by tracking the ejection and attachment of pipette tips. This will necessitate algorithmic work that will test our ability to rapidly add new objects into our framework and to track the separation and merging of objects.