Multi-view Object Detection with Neural 3D Scene Understanding

Detection and classification of vehicles from sensor data obtained via airborne platforms is an imperative capability due to its potential applications in recognition, tracking, and targeting. Obtaining this capability is highly nontrivial in occluded urban scenes due to the sheer variety of possible occlusions. In these settings, to build robust detection and classification pipelines, we need more information than is provided by a single electro-optic camera. We propose a novel pipeline for multi-view object detection and classification based on neural scene understanding. Our approach explicitly models 3D scene geometry and semantics, while retaining the modeling flexibility and end-to-end differentiability of state-of-the-art 2D image fusion pipelines. We will train and evaluate this multi-view detection model on MVMO, a rich synthetic dataset containing wide-baseline multi-view images and semantic segmentations for scenes containing household objects, and VSAI, a smaller dataset containing multi-view images and bounding box annotations for urban scenes with partially occluded vehicles. For simplicity, we assume known sensor poses, static scene content across views, no latency requirements at inference time, and access to GPU compute for training. We also describe how to remove the assumptions of known sensor poses and unlimited inference latency in an operational prototype using additional system components.