Object Recognition of Imagery Entities (ORIENT)

Under this Phase II effort, entitled Object Recognition of Imagery ENTities (ORIENT), Language Computer Corporation (LCC) will leverage our 20+ years of experience developing ML technologies for commercial and government interests – in such diverse fields as video- and image-based scene analysis, image categorization and decomposition, generative AI, knowledge engineering, and question answering – to develop an interpretable, fine-grained object detection pipeline capable of quickly distinguishing hundreds of different types of objects from diverse visual data sources without the need for extensive annotated training data or human-in-the-loop (HITL). To this end, we will expand upon our existing state-of-the-art image processing technologies to leverage latent object knowledge in support of zero-shot (or few-shot) categorization of detected objects with a focus on electro-optical (EO) satellite imagery, synthetic aperture radar (SAR) satellite imagery, and ground-based EO and infrared (IR) imagery. We divide our research into four tasks. Under Task 1, we will develop an ensemble of probabilistic and self-supervised techniques to identify all objects in a scene under the “open world” paradigm. This will allow us to cast a wide net at ingestion time and to leverage existing open-source datasets for coarse-grained object detection. In Task 2, we will employ a suite of large language model (LLM) prompts in a multi-stage dialogue to surface fine-grained knowledge of each of the program object-of-interest (OOI) classes for incorporation into a collection of probabilistic object profiles. In particular, we will focus on identifying which features of those objects are likely to be detectable (within a particular imagery type) – including their parts and any visible attributes of those parts (e.g., color, size, shape, material, markings). The features defined in these object profiles will then be exploited under Task 3 to develop a portfolio of lightweight feature detectors trained indirectly using publicly available images via text-to-patch retrieval in a fine-tuned foundational model embedding space. Task 4 will be focused on the fine-grained classification of all detected objects using Pre-Weighted Neural Feature Trees (P-NFTs) that iteratively test an image for relevant features using the Task 3 feature detectors. These P-NFTs, which combine the expressiveness of deep learning models with the explainability of decision trees, will be trained using synthetic feature data constructed by intelligently sampling feature information from the Task 2 object profiles along with features from all imagery data made available in Project Linchpin. We will focus throughout this project on integration and deployability, making sure that all developed software and models can be easily delivered, installed, configured, tested, and deployed within Project Linchpin’s AI Unclassified Operations Environment, as well as other relevant Army and DoD platforms.