Enhancing Information Density to Improve Remote Data Access and Reduce Costs: Integrating OpenVisus Streaming with ZFP Compression

This STTR/SBIR project aims to integrate ASCR-funded ZFP compression software with OpenViSUS, enhancing large-scale data management for scientific research. OpenViSUS, a framework for fast data reading and writing, is widely used by scientists in DOE labs, Cornell High Energy Synchrotron Source, and NASA for data storage, visualization, analysis, and sharing. The project addresses the challenge of managing petascale datasets produced by modern scanning modalities. Currently, OpenViSUS provides interactive access to massive remote data through efficient access to subsets and resolution levels. By integrating ZFP compression, we aim to significantly improve data density, which has the dual benefits of accelerating information throughput and reducing storage costs. Initial tests show promising results, with ZFP compression reducing a 400TB NASA dataset to 112TB at 16-bit precision, achieving a 3.57 compression factor with low error. ZFP offers various compression modes, levels, and block sizes, allowing fine-tuned control over data precision and accuracy during encoding and decoding. Our proposed integration will require the development of algorithms to map user-facing requirements, such as precision, accuracy, or bandwidth needs, to actual costs. The system must determine optimal compression schemes, automating the complex and highly flexible encoding/decoding parameter selection. The compression scheme will operate on the back-end, hiding the complexity; to a user, massive remote data will still appear accessible via standard formats and integrate easily with existing workflows through web portals/dashboards, Python, Jupyter Notebooks, C++ library calls, or visualization tools. Key development areas include criteria and algorithms for network/cost configuration, user priorities and cost models, code hardening and ease of integration, and adaptation for cloud, HPC, and other computing environments. The project's broader impact includes revolutionizing data management practices across DOE labs and partner institutions, accelerating scientific discovery by reducing the time and cost of handling massive datasets and enabling FAIR principles for massive data. It will encourage cross-disciplinary applications and stimulate technological innovation and private-sector commercialization. This project aligns with DOE's mission to address diverse energy, environmental, and scientific challenges through transformative technology solutions. It has the potential to significantly advance DOE's research capabilities and contribute to America's security and prosperity.