An Open Analysis Platform with In-Situ Machine Vision for Electron Microscopy

Microscopy workflows can include a detailed image analysis and quantification effort that has proven to be time consuming, cost ineffective, and potentially biased. Recent developments in machine learning and artificial intelligence methods have signaled a revolution in these efforts leading to accelerated and consistent quantification of digitally generated microscopy images. Given this, these innovative methods typically require cloud computing resources and at least some knowledge of software development and computing programming. This is further exacerbated by the seemingly daily release of new machine learning methods. An open software-hardware platform that overcomes these issues and embraces the rapidly evolving field has surpassed the feasibility stage and will be developed for full-scale commercialization. The technology, which uses a web-based interface hosted on a graphical processing unit-equipped edge computing device, was successfully demonstrated to host and run a state-of-the-art machine learning model to enable the first augmented reality in-situ transmission electron microscopy experiments for irradiated materials development and qualification. This technology will be further advanced to host a range of additional machine learning techniques while providing a flexible platform for integrating future techniques and methods for any automated, real-time image analysis and quantification effort. It has already been demonstrated that the interactive, real-time platform at the heart of the technology can augment the features of advanced electron microscopes resulting in reduced person hours (and in turn costs) within microscopy workflows. The proposed efforts will then extend past the niche application demonstrated in the initial feasibility study by leveraging the agnostic nature of the communication protocols used for interfacing with microscopes. Specifically, the platform’s edge computing device and user interface software stacks will be advanced through concentrated efforts on expanding the hosting, deployment, and dissemination of current and future machine learning methods to enable widespread adoption in industry, energy, and medical sectors using optical, charged particle, and scanning probe microscopes. The resulting outcome will be a flexible platform that reduces labor costs by up to 80%, decouples expertise in routine microscopy tasks, and accelerates the innovation lifecycle of new technologies that rely on quantitative microscopy, such as technologies and products under development in material science, biology, medical imaging, geology, energy storage, and energy consumption.