An Image Fusion and Data Analysis Tool for Multiple Microscopies

In this effort, Toyon will address the problem of registering and aligning images from multiple different microscopy modalities. Different types of microscopes give different information about a material sample and combining insights from these is necessary to completely characterize a sample. Due to large differences in field-of-view and spatial resolution, as well as different detection physics and data types, it is very difficult in general to align images from different types of microscopy. Examples of modalities that Toyon will develop software to co-register include SEM/TEM, X-ray/electron diffraction, and spatially-resolved spectroscopies such as EELS, EDS, and Raman. Our automated approach will greatly improve on the existing solution where highly-trained technicians do this alignment manually, which is time-consuming and expensive.Toyon will adapt an existing robust statistical registration algorithm developed originally for remote sensing and geospatial applications. We will develop pre-processing transformations for each data type to allow our adapted algorithm to perform well. Toyon will validate our approach using data collected from a variety of physical samples simultaneously scanned with STEM (bright and dark fields), EELS, EDS, and 4D-STEM (diffraction). Being simultaneous scans, these data can serve as a ground truth comparison for our registration algorithm to validate our approach before we generalize to images taken with different machines and a broader array of techniques.In Phase I, Toyon will collect the simultaneous-scan sample data for use in validation, develop appropriate pre-processing transformations, adapt our registration algorithm to this task, and build a user-facing application to display the results of the registration process. The post-processing research and algorithm adaptation will proceed in parallel to allow development of a performant end-to-end pipeline. The user-facing application will provide post-registration fused images of different modalities as well as statistical summaries of viewed sample regions (e.g., elemental composition, crystal phase, etc.)Commercial applications are numerous since many industries utilize the microscopy technologies we will research. The semiconductor manufacturing and oil&gas industries use SEM paired with other microscopies to characterize, respectively, their products or the rock matrices that hold the materials they extract. Biotechnology research, including drug development and disease characterization, often require multiple microscopy techniques on the same physical sample. Potential customers include companies in these industries as well as academic research institutions. By making multi-modality complete sample characterization faster, cheaper, and more accessible, we aim to facilitate the collection of massive datasets that can be used for deep learning research to allow discovery of subtle correlations and relationships only discernable once sufficiently large datasets are assembled and made available for learning algorithms.