Generative Unbiased 3D Semantic Segmentation

3D models are commonly generated from both multiview satellite and FMV sources using photogrammetricmethods. However, such models lack semantic labels (e.g. segmentation of buildings, roads, vegetation,vehicles, etc.) needed for further analytics. Prior work relies largely on discriminative models to classify3D surface points using local context, but discriminative models have difficulty learning complex relationshipsbetween objects that generalize to new domains. Recent work in generative models, such as StableDiffusion, trained on a combination of imagery and language have shown great power in modeling andgenerating large scale imagery in a consistent, meaningful way. It has also been shown that generativemodels used in semantic segmentation of images can generalize better to new domains than discriminativemodels. Kitware proposes to leverage these recent findings to provide semantic segmentation of 3Dsurfaces via fusion of generated 2D segmentation maps. We will leverage the fact that these 3D models arederived from 2D imagery to adapt and finetune existing image generative models, like Stable Diffusion, togenerate semantically meaningful segmentation in 2D before fusing into consistent 3D surface labels. Theresult will be 3D segmentation and detection results that generalize far better to new environments.