Supporting Sparse Data in HDF5

Sparse data is common in many scientific disciplines. Examples include large-scale simulations of physical phenomena, High Energy Physics experiments, machine learning applications, and many more. Acquired data is stored in a scientific data format that became a de facto standard for data management in government, academia and industry. As the amount of data in the scientific format continues to grow due to higher instrument and detector resolution, higher sampling rates, etc., there is a clear demand for efficient management of sparse data. The support for sparse data is also accompanied by a growing demand in the experimental sciences to perform data analysis in the high-performance environment, where the scientific data format is widely used. Adding support for sparse data will eliminate demand for custom data processing software and will reduce the size of required storage and memory usage for applications that work with sparse data. The project will implement sparse data storage in the scientific data format and will make necessary changes to the software with no disruptions to the applications that use the format. Access to sparse storage will be transparent to the applications and will not require special sparse data encoding or additional coding effort. We will use existing elements of the software to implement the new feature while enhancing some of the software components and documentation. In Phase I of the project, we will design and prototype sparse storage and will update existing software components to provide basic functionality. The software prototyped in Phase I and refined in Phase II will: Deliver sparse storage functionality to the experimental and modeling community. Deliver a specialized Toolkit for high-speed data acquisition to manage sparse data in the scientific format. We will improve the robustness of the open source software by our contributions to the library, test suites and documentation. Discussions with customers have identified the need for sparse data storage functionality amongst the scientific and engineering communities that use scientific data formats in high-performance and experimental environments. The proposed solution is uniquely able to address this community need.