Rapid Multi-Modal Microscopy Feature Correlation Tool (RM-MFCT)

One of the most difficult challenges faced to combine microscope information is the relocation of particulate matter or region of interest from one instrument platform to the other. This is often due to differences in morphological appearances of the particles and magnification changes between the scanning electron microscopy and the optical-based instrument. In addition, skew effect caused by optical aberration, sample movement and system drift also pose challenges for the feature correspondence work. Develop and design a novel rapid multi-modal microscopy feature correlation tool combining traditional computer vision techniques with Deep Learning, which is anticipated to provide quick and accurate registration performance while addressing region of interest relocation, correction of aberration effect, cost requirements, and the ability to meet multiple Concept-of-Operations. The rapid multi-modal microscopy feature correlation tool algorithm pipeline has been tested on 1,350 image pairs collected using three modalities, visible and near-infrared camera, Short-wave infrared camera and visible light camera. The overall correction success rate is 90.4% and the overall correlation error is within one pixel (0.7 pixel in average). In addition, the proposed algorithm pipeline is able to address skew effect due to optical aberration. Based on the success of this Phase I work, Phase II will consist of refining the rapid multi-modal microscopy feature correlation tool notional design, developing more robust correlation algorithm, fully transitioning the algorithm pipeline to a software application, and conducting the testing work on various of microscope data. Developing and testing this software application is the first step towards finalizing a set of requirements that can guide the commercialization for microscope software market. The resulting product of the Micro-Core project will deliver value immediately to the forensics community, then more broadly to machine vision market, the image processing market, and the image recognition market. Micro-Core will automate the process of image registration that is a necessity in any process that takes images from more than one camera or sensor. Micro-core will improve the speed, the accuracy, and reduce the need for human intervention in image- registration processes.