TeleSculptor: Semantic 3D Scene Modeling from High Zoom Reconnaissance Video

ABSTRACT: 3D model generation from long-range ISR video is an important part of Special Forces operation planning. The current workflow is highly manual and time consuming. Existing automated algorithms for 3D reconstruction from high- and variable-zoom FMV are fraught with failure due to both practical issues (e.g. metadata handling) and algorithm limitations that result in poor quality models. Kitware proposes TeleSculptor: Semantic 3D Scene Modeling from High Zoom Reconnaissance Video. Telesculptor builds on Kitware's established software tools (Burn-OUT, MAP-Tk, Super3D) that provide per-frame metadata burn-in detection, erroneous KLV handling, camera calibration for high-zoom FMV including planar scenes, and dense surface estimation. Telesculptor breaches the algorithmic barriers by incorporating semantic reasoning into 3D reconstruction. It will segment the video frames into semantic regions like ground, wall, door, window, and vegetation using the latest advances in deep learning. It will use these image segmentations to jointly estimate a 3D surface and a segmentation of that 3D surface into semantic categories. The semantic labels allow local geometric constraints that improve 3D model accuracy. Our proposal will enable fully automated reconstruction of high quality 3D models, saving countless man-hours. It will enable 3D exploitation of hours FMV that would otherwise sit idle in the archives.; BENEFIT: The primary benefit of the proposed work is a solution to problem of automated 3D scene reconstruction from high zoom aerial reconnaissance video. Our software will directly benefit mission planning at AFSOC, for example. In the time it takes one user to build a model from video, our system could build thousands of models from thousands of videos automatically. This could change the way video is collected and used for mission planning. In addition to military applications, the same technology applies to commercial and humanitarian applications in the budding commercial UAS market. 3D models from video can be used to measure material displacement in mining and damage in disaster recovery operations.A secondary benefit of the proposed work is that all of the developed technology will be releasedas open source. The existence of the open source solution will benefit the research community, and the research community will, in turn, enhance the software at no cost to the program. The open source toolkit will also directly benefit Kitware by creating more consulting contracts surrounding the technology.