Text supervised 3D vision-and-language representation learning for zero-shot geospatial analytics

Stottler Henke proposes to build the resource-efficient TESSERA system, a novel end-to-end approach for learning 3D representations from multi-view satellite imagery and full motion videos (FMV) for downstream tasks relating to 3D Geospatial Analytics.  TESSERA operates by ingesting a variable number of multi-view images and producing a labeled point cloud from an open-vocabulary set of classes.  This system's key innovation lies in its ability to train on a purely text-supervised basis, wherein the text, extracted through state-of-the-art (SOTA) Generative AI models, encapsulates the numerous spatial and temporal relationships within the scene to facilitate open-vocabulary zero-shot transfer learning to an array of downstream geospatial analysis tasks, including semantic segmentation, detection, and classification.  The proposed pipeline entails 2D feature extraction, 3D reconstruction, and language-image-point cloud joint processing for a versatile, text-supervised architecture that opens up new avenues for streamlined and zero-shot transferrable geospatial analytics tasks.This effort builds upon Stottler Henke’s experience and existing technological foundation for remote-sensing image processing and natural language processing.  Phase I will demonstrate the superiority of our approach over a purely discriminate multi-task baseline ML architecture, and it will pave the way for full-scale system development and evaluation during Phase II.