Pan3D: Open Source Scalable and Reproducible Scientific Workflows for 3D Data Analytics

C53-01a-271272The problem to be addressed arises from the recent and rapid growth of large three- dimensional (3D) and time-varying datasets produced from sensor/observation systems, numerical simulation, and AI models and the current difficulties incorporating these data into research workflows in many data science communities. To address this problem, Pan3D will fill critical needs for (i) data interoperability between modeling (numerical, AI) and visualization systems, (ii) easy-to-use visualizations for web- and Jupyter-based workflows, and (iii) software platforms integrating these features to provide interactive analytics environments for both cloud and high-performance computing (HPC) workflows. The Phase I effort established technical feasibility by successfully prototyping two software capabilities. The first Phase I prototype produced a working interface between standard software for labeled NumPy arrays (Xarray) and standard software for 3D visualization (VTK). The second prototype demonstrated the visualization of 3D datasets in Jupyter notebooks with both "back end" and "front end" rendering capabilities (the latter being important for JupyterHub and other distributed-computing environments). In Phase II, the data interoperability software will be reorganized and revised for better modularity and more seamless integration with visualization pipelines. The visualization prototype will be streamlined in several ways that include refactoring out unnecessary code associated with pre-existing software components used in Phase I. The resulting software products will be demonstrated with desktop workflows and will also be integrated with cloud and HPC systems to demonstrate their benefits in collaborative environments at scale. The Phase II results will significantly improve scientists' productivity and efficiency by enabling them to focus on data interpretation, analysis, and model development. The open, modular platform to be developed in Phase II will provide significant commercial opportunities to Kitware through customization and other professional services, leveraging the business model Kitware has successfully used since its founding over two decades ago.