Open Source Platform for Live Processing of High Data Rate Electron Microscopy

4D scanning transmission electron microscopy (STEM) is experiencing a revolution both in terms of data size, and data veracity, as are many other areas of experimental data acquisition. This presents enormous opportunities to make new scientific discoveries, but is also hamstrung by aging software that is no longer fit for purpose. Experimental data acquisition platforms must make the same investments in software infrastructure that the simulation community made decades ago to take advantage of high-performance computing (HPC) if they hope to realize the full potential of the latest advances made in detector technology. This requires rethinking the design and deployment of hardware, moving away from small, fixed commodity computing capabilities with closed interfaces to open, modular, scalable systems designed for parallelism and throughput. The technical approach will leverage best-in-class open source technologies, and develop new approaches strategically to build an open source platform for experimental data acquisition targeting high data rate electron microscopy. The Phase I project will develop a software platform that utilities the power and performance of modern HPC resource to ingest the large volume of data produced by electron microscopes using direct electron detectors. A set of configurable pipelines will be used to enable the processing to be tailored to the particular problem being investigated. It will focus on a “live” pipeline to perform simple processing that can display an image of the current state of the microscope. This pipeline will also process data, and package it in a structured binary format. A triggered pipeline will use the queuing system on high-performance computing resources to do further analysis. Visualization of the live and processed pipeline will be offered through a web application requiring minimal desktop resources. This project will leverage investments made by the DOE and other agencies to provide a powerful software platform for the experimental data community working with latest electron microscopy, such as 4D STEM detectors. It will create a platform that can serve as a reference for the com- munity. The use of permissive open source licensing will make the platform available to all, with a simple deployment strategy offering the ability to keep data close to where it is generated. This advanced software platform will require customization for each deployment, creating opportunities for software services that are typically offered to extend open source software platforms.