An Open Analysis Platform with In-Situ Machine Vision for Electron Microscopy

Recent increases in data collection hardware over the past several decades have resulted in the observation that most data-rich microscopy workflows are now software limited. Machine Learning and Artificial Intelligence methods could eliminate many of the software bottlenecks by performing automated data analysis and image evaluation. Given this, the adoption of these advanced methods has been hampered by the lack of combined hardware-software platforms that make their use ‘turn-key’ in common microscopy workflows. Theia Scientific has developed a platform that eliminates the barriers to entry for using Machine Learning and Artificial Intelligence for routine and advanced microscopy activities. The technology leverages a modern web application software stack and Graphical Processing Unit-equipped edge and nearedge computing devices to host and run state-of-the-art Machine Learning models. Previous efforts have enabled the technology to perform data collection and analysis with the user observing the results as effectively a passive user experience. This observability technology will be further advanced to provide a new user experience that incorporates interactivity into the data collection, analysis, and manipulation pipelines. The interactivity, or active applications, will be developed based on commonly requested features from microscopy user groups. The proposed efforts will then provide a rich user experience within a modern user interface that enables the technology to extend past niche applications previously demonstrated in prior efforts. Specifically, object tracking while using custom tiling-based inference algorithms between sequences of images, bidirectional instrument communication with feedback control, and automated smart instrumentation and image analysis parameter selection and optimization will be implemented and demonstrated in-situ electron microscopy. The resulting outcome will be a flexible platform that reduces labor costs by up to 80%, decouples expertise in routine microscopy tasks, and accelerates the innovation lifecycle of new technologies that rely on quantitative microscopy, such as technologies and products under development in material science, biology, medical imaging, geology, energy storage, and energy consumption.