GD3D - Generative AI for Tactical 3D ISR

There is an increasing need to automatically extract land features, roads, curbs, building contours, from 3D data either derived from images or video. This task predominantly involved human interpretation of the 3D data. Recent advances of AI, however, enables less and less human interaction by achieving higher and higher performance on semantic segmentation and object detection tasks. The emerging machine learning models, in particular, large language models (LLMs) are achieving human-like performances on many tasks and making headlines in the national news. In our proposal, entitled Generative and Discriminative AI for Automated 3D Segmentation (GD3D), we want to use network components that power LLMs to improve on the state-of-the-art for 3D data segmentation and object detection. Therefore, we propose an end-to-end multi-head deep neural network composed of a generative and a discriminative neural network. The generative network in the form of a language model takes the scene and generates outputs which describe the objects and their relationships on the scene. Then, a discriminative network takes these relationships and fuses it with the 3D data input to produce labeled data and object bounding boxes. It is expected that the discriminative network will achieve improved performance by having access to the object relations. LLM training requires large datasets and long computation times at a scale which is not affordable for a small business. Therefore we take another approach where we use pre-trained foundational language models. Although training on our specific task is still required for refinement, this significantly decreases the cost of the development. The drawback is that we cannot fundamentally change the domain where these networks are designed for and trained on. Language models that generate text inputs almost exclusively require image inputs. Therefore, we need to convert the 3D data to 2D structured inputs. We will use various 2D representations such as depth maps, normal maps, ortho image, or DEMs. These 2D representations of the scene will be the input of the generative model and the discriminative model.