A Scalable Platform for Real-Time Microscopy Image Analysis Using Artificial Intelligence and Machine Learning

Materials sample characterization by electron microscopy with multiple modes of detection provides more robust and information-rich data, but its adoption remains limited because of the dramatic increase in off- the-microscope data analysis work. Essentially, the necessary, but labor-intensive post-acquisition data analysis for multi-modal microscopy has kept it a low throughput technique that is completed off- microscope. The two primary contributing factors to the post-acquisition labor are (i) poor integration of post-acquisition image analysis systems and (ii) low interoperability between proprietary control software for each detector connected to a microscope. An integrated, platform-agnostic system with automated, concurrent image analysis and quantitation for multiple detectors will be developed. Such a system will enable real-time multi-modal microscopy image analysis, aggregation, and registration at the point-of- acquisition with output displayed as a set of layers over the microscope control software. Real-time electron microscopy image analysis for a single detector has already been demonstrated using a web-based software stack, recent advances in artificial intelligence, and a graphical processing unit-equipped edge computing device. The proposed technology will scale the established single detector image analysis platform to run concurrently for multiple detectors to enable real-time multi-modal microscopy. Success will be demonstrated by deployment of a fully integrated real-time multi-modal microscopy image analysis system within a Nuclear Sciences User Facility microscopy center. It is anticipated this system will transform the laborious, manual multi-modal microscopy image analysis workflow into being identical to the scalable, automated real-time single detector-like workflow and user experience. This transformation will enable wider adoption of multi-modal microscopy for materials discovery and qualification based on characterization tasks with additional growth into other forms of microscopy, such as optical and X-ray for biology and medical applications.