Cloud-based 3D Instance Segmentation for Expeditionary Airfield Damage Assessment

With the proliferation of 3D capture technologies such as smartphone LIDAR sensors, depth cameras, neural radiance field (NeRF) imaging, amongst others, and ever-improving cloud compute capabilities, it is nowápossible to train computer visionámodels to perform cloud-based instance segmentation. The objective of cloud-based instance segmentation is toádetect every uniqueá3D objectábased in an indoor or outdoor environment, directly from a web browser or mobile application. Currently, most conventional 3D point cloud instance segmentation architectures inherentlyáachieve object detection and semantic segmentation as a precursor to instance segmentation. For this reason, and because instance segmentation provides a greater resolution of information thanáobject detection alone, this SBIR Phase One proposal will explore the feasibility to train machine learning models to achieve an instance segmentation architecture that can be deployed to the cloud. If successful, this ensures that users can extract precise information at a point-wise level from a 3D-scanned indoor or outdoor scene on any web-enabled device. á The first step of this proposal is to explore and choose the appropriateásegmentation model for the purpose of identifying objects of interest in an airfield damage assessment scenario.áUsing a sparse 3D backbone resembling a deep neural network such as U-Net or PointNet architecture, we canáextract features from the original 3D point cloud and apply permutation invariant transformations. The decision to use sparse convolutions is due to the observation that 3D point clouds are typically sparse data structures to begin with. Next, Stitch3D will take the output features from the backbone architecture and apply them asáinput to 1) a local feature extractor architecture that extracts features at the local resolution per point in the point cloud,áand 2) a global feature extractor architectureáthat transforms the same input from the backbone to a set of instance-wise features. Finally, we will determine the feasibility resulting from the outputs of the local and global feature extractor architectures as inputs into the main instance segmentation network. If successful, research resulting from this SBIR Phase One effort will allow users to perform real-world instance segmentation on any indoor or outdoor 3D-scanned scenes. Specifically, in the context of expeditionary airfield damage assessment and repair, engineers and airfield managers willábeáable to identify different types of damaged aircrafts or unexploded ordnances in the aftermath ofáan airfield attack from an aerial LIDAR scan.