Supporting Sparse Data in HDF5

Sparse data is common in many scientific disciplines. Examples include large-scale simulations of physical phenomena, High Energy Physics experiments, machine learning applications, and many more. Acquired data is stored in an open-source scientific data format that became a de facto standard for data management in academia and industry. As the amount of data in the scientific format continues to grow due to higher instrument and detector resolution, higher sampling rates, etc., there is a clear demand for efficient management of sparse data. The support for sparse data is also accompanied by a growing demand in the experimental sciences to perform data analysis in the high-performance environment, where the scientific data format is widely used. Adding support for sparse data will eliminate demand for custom data processing software and will reduce the size of required storage and memory usage for applications that work with sparse data. The project will implement sparse data storage in the scientific data format and will make necessary changes to the software with no disruptions to the applications that use the format. Access to sparse data storage will be transparent to the applications and will not require special sparse data encoding or additional coding effort. We will use existing elements of the software to implement the new feature while enhancing some of the software components and documentation. In Phase I of the project, we designed and prototyped extensions to the scientific data format to support sparse data storage along with the required changes to the software. We made our designs extensible to support efficient storage of variable-length and non-homogeneous data. These extensions are highly anticipated by the users and can be implemented in the future without any modifications to the file format. We reviewed the proposed changes with the users and maintainers of the open-source software and assured support for our approach. In Phase II we will deliver sparse storage functionality in the software and we will implement high-speed data acquisition mechanism to manage sparse data in the scientific format. The benefits of our work are two-fold: Efficient support for sparse data storage opens new opportunities in many research areas including material sciences, genomics, biomedical research and clinical investigations. Our solution removes performance and storage bottlenecks shortening time to solutions to the critical problems. Our contributions will improve the robustness of the open-source software.