Multi-Task Scale -aware Continuous and Localizable Embeddings

In Phase I, our team of Kitware and UC-Berkeley developed Scale-MAE by adding ground sample distance (GSD) to positional encodings, and produced a multiscale representation that achieves state-of-the art results across image classification, semantic segmentation, and object detection tasks. In Phase II, we will create a remote sensing pretraining toolkit to enable fast and easy experimentation with multiple self supervised pertaining techniques that create foundational deep neural network models applicable across NGA. The foundational networks will be tested on the tasks as Scale-MAE in Phase I, and we will also benchmark performance for key point matching. Phase II will extend our Scale-MAE work by integrating additional metadata into the network to increase accuracy by providing it with more information; we will extend the approach to handle inputs including NTM, multi-spectral, and SAR data. Finally, the downstream task networks will be transitioned into NGA SAFFIRE for integration and evaluation. This system will enable NGA to quickly train and deploy new detectors to quickly respond to shifting needs and reduce the time from the analyst’s demand for a new task capability to the execution and availability of said capability.