Spatial Registration of Multi-Sensor Data Using 3D Reconstruction

In current and future military applications, multiple sensing technologies are being employed to provide accurate situational awareness, resulting in successful and safely-deployed missions. Many threats, including mines and improvised explosive devices (IEDs), are intelligently hidden from human sight, but can be successfully detected using a combination of ground penetrating radar (GPR) and various IR and visible imaging sensors. Toyon, in collaboration with Prof. Joseph Mundy of Brown University, proposes to design and develop a system capable of autonomously performing multi-image spatial registration across a number of sensor modalities. Our approach will accurately co-register all sensors during a single mission, and if desired, across missions, using un-surveyed incidental 3D control points that occur naturally in the environment and which are automatically detected and tracked by the registration software. Information obtained from detecting and tracking the control points and from estimates of the vehicle’s relative motion between scans will be used to perform high-fidelity joint automatic sensor calibration and 3D reconstruction for each sensor in a common vehicle coordinate frame, thereby registering the data. The data registration will then be optimized jointly across sensor modes by maximizing a mutual information metric, with the goal of achieving sub-pixel registration accuracy. The registered data can be presented to the system user or for further processing in the form of registered 3D image cubes, or 2D images registered in one of the sensor image planes or from a novel viewpoint. In either case, a multi-sensor multi-dimensional registered data set is provided for exploitation, e.g., for mine or IED detection. In Phase I we will demonstrate the feasibility of the proposed technical approach, evaluate the multi-sensor spatial registration accuracy obtained with real-world data collections in complex scenes, document the results, and provide recommendations for further research and development in Phase II and for commercialization of the developed technology.