Open Real-Time Access at the Edge for Ultra-Fast EM Data

High-speed direct detection cameras have enabled a variety of experiments to probe structural dynamics in response to various stimuli. As microscopy equipment and methods improve, it is increasingly necessary to perform specialized image processing to generate interpretable results from the gluttony of acquired data. At present, the vast majority of this processing occurs after data acquisition has been completed, precluding the user from monitoring the specimen in real-time and hindering automation via machine learning. This project aims to develop an open platform for real-time processing of fast direct detection camera data by enabling user-access to GPU or FPGA computing resources. During Phase I of this project, we began investigating and implementing a prototype system enabling open, direct, real-time access to high-speed electron microscopy camera frames in GPU memory. To prevent competition for computing resources with data acquisition, this direct data access was facilitated by an optimized interface connected to a second Linux computer, where users can implement their own real-time image processing functions without risking lost data on the data acquisition computer. During Phase II, we will expand this proof-of-concept to use next-generation hardware to maximize its bandwidth and ensure it can keep up with future high-speed cameras. We will also develop an open application programming interface (API) and software development kit (SDK) to enable third-party development and ensure the system is open for use by the broad scientific community. While this system will enable third-party development on a relatively easy-to-use Linux-based platform, and it will provide ample computing power at the edge for a wide variety of user-defined algorithms, the potential disadvantage of this system is latency, which we estimate to be on the order of tens of milliseconds. Although this latency is trivial for most applications, it may hinder the development of some techniques requiring significantly lower latency, such as intelligent beam blanking to minimize specimen radiation damage. We therefore propose to expand on our Phase I project by also implementing a similar real-time ultra-low-latency system for direct access to camera data on FPGAs included in the camera framegrabber card. In Phase II, we will implement a new framegrabber for our cameras that includes a user-programmable FPGA, and we will similarly develop an API and SDK for third-party implementation of real-time FPGA-based data analysis. At the conclusion of Phase II, we expect to offer two commercialized systems for open, real-time data processing in electron microscopy: one with GPUs with ample computing power and low learning curve, and the other with FPGAs for ultra-low latency but with limited computing power and a steeper learning curve.