Streamlining Volumetric Imaging, Analysis and Publication Using Immersive Virtual Reality

Over the past 15 years, new imaging technologies and methods for high throughput imaging have revolutionized structural biology by extending the resolution and scale of collected images in 3 dimensions. The resulting image volumes are more typically hundreds of GB to even tens of TB and in some cases approach PB sizes. These file sizes pose challenges for image acquisition, image analysis, and communication of a representative set of raw data and quantification. Image acquisition runs can be lengthy and expensive, and often errors are not identified until after the completion of scanning. Large files contain many structures, and require machine learning (ML) strategies in a context that permits error correction. Scientific communication requires tools for ready access to raw data, and more efficient methods to communicate the rapidly accumulating sets of scientific information. We propose to leverage virtual reality (VR) and verbal communication within the VR environment, to streamline each of these stages of scientific work, by capitalizing on the more natural abilities for stereoscopic vision and hearing to process scenes and language. Based upon the tool base and direct volume rendering of large files that we have established in our VR software, called syGlass, we will first integrate VR into the microscope controls for tuning the microscope and then efficiently inspecting images in 3D as they are acquired (Aim 1). Next, we will introduce novel domain adaptation techniques in the ML field to scale up 3D image quantification capabilities for current acquisition sizes, by coupling them with user-optimized experiences that do not require ML expertise, and yet provide automated and accurate results (Aim 2). Finally, we will provide tools to efficiently generate narrated scientific presentations in VR for use in the lab setting, as manuscript publications, and for production of educational materials (Aim 3). In each of these activities, we will introduce paradigm shifts in the management of experiments, analysis of the resulting data, and publication of manuscripts and materials to other scientists and the general public.The goal of this project is to speed the pipeline from image acquisition to communication of analyzed data for large image files (big data). We propose to leverage virtual reality to change the way users interact with their microscope, provide new methods for more accurate quantification and make scientific data more transparent, and more accessible to specialists and the general public. These new paradigms are applicable to basic, pre-clinical and clinical research, and serve the goals of big data projects to generate more reliable and encompassing scientific conclusions.