An Open Analysis Platform with In-Situ Machine Vision for Electron Microscopy

Electron microscopy image analysis workflows are currently a non-scalable, biased process that requires extensive time and expertise. Attempts to generate a scalable and non-biased automated image analysis workflow using artificial intelligence and machine learning technologies at the point-of-acquisition have been thwarted by (i) limited, or impossible, access to external cloud computing resources and environments, (ii) lack of a consistent, streamlined end-user experience for distribution and deployment within these network-constrained environments, and (iii) poor interactivity of real-time image analysis results in closed electron microscope system software. An open software-hardware platform that overcomes these deficiencies will be developed and include capabilities for displaying real-time image analysis results and machine learning output as a live dashboard and overlay in microscope control software. The proposed technology, which uses a web-based interface hosted on a graphical processing unit-equipped edge computing device, has already been demonstrated to overcome the currently identified issues through internal proof-of-concept efforts. This proof-of-concept will be driven to commercialization by incorporating a container-based machine learning model deployment framework and optimizing the communication pipeline between the microscope and edge computing device. Success will be demonstrated and evaluated by creating the first ever augmented reality electron microscope running a community driven automated feature detection algorithm. It is anticipated the interactive, real-time platform, which augments the feature set of any digitally controlled microscope, will result in an 80% reduction of person hours associated with labor intensive, time consuming image analysis tasks, while providing reproducible results unbiased by human-based imaging detection, classification, and quantification tasks. The proposed flexible platform ensures applicability to a range of microscopy workflows including material science, biology, medical imaging, geology, and hyperspectral imaging with significant impact in high-use equipment, such as those residing in government-supported interdisciplinary research centers and vision-based industrial quality control centers.