Direct Real-Time GPU Access for Ultra-Fast Electron Microscopy Data

C55-10a-270599High-speed direct detection cameras have enabled a variety of experiments to probe structural dynamics in response to various stimuli. As microscope stages improve, it will become possible to capture these dynamics in three dimensions (3D) by repeatedly acquiring rapid continuous rotation tomography data. However, such data requires complex image processing to generate interpretable reconstructions of the specimen from the gluttony of acquired data. At present, the vast majority of this processing occurs after data acquisition has been completed, precluding the user from monitoring the specimen in real-time and hindering automation via tools such as machine learning. While a few national laboratories have successfully developed systems to transfer data in real-time to on- site supercomputing resources for processing, such solutions are untenable for the vast majority of TEM labs. To resolve these challenges and enable real-time reconstruction and feedback, we propose to provide direct access to high-speed camera data in GPU memory, in a format that can be readily used by custom image processing algorithms. To prevent competition for computing resources with data acquisition, this direct real- time access to camera data in GPU memory will use a second computer connected through an ultra-high- speed interface to the camera computer. An open API will be developed to enable community development. During Phase I, we will develop and demonstrate an application programming interface (API) that enables direct access to processed frames from our DE-16 camera system in shared memory on an Nvidia graphics card (GPU) in the camera computer system. Additionally, we will develop a prototype system in which data from the camera computer can be transferred to a GPU on a second computer. The bandwidth and network transfer overhead of this prototype system will be evaluated to prepare for expansion, optimization, and commercialization of this system in Phase II. The resulting add-on GPU-based platform will enable immediate feedback for complex electron microscopy experiments such as continuous-rotation tomography, which is invaluable to microscopists as they investigate dynamic specimens. Perhaps even more significant, this system will also provide a platform for development of new real-time algorithms and advanced electron microscopy techniques by the broad scientific community