Toward multi-threaded concurrency in HDF5.

C53-02a-271303Open-source standard scientific software and hierarchical data format for data management is the de facto standard for scientific and engineering applications. It is widely used in large-scale simulations of physical phenomena, aircraft flight testing, high-speed neutron and X-ray detectors data acquisition, and many more. One of the known limitations of the software is a lack of multi-threaded support for accessing data. This creates adoption, deployment and performance barriers for multi-threaded applications, especially for machine learning applications. To date, no effort has been made to overcome this restriction, mostly due to the difficulty of retrofitting a large code base with thread concurrency. Recent developments in the architecture of the software allow us to suggest a strategy of retrofitting thread-safety in an incremental way without affecting any other development going on in the software and without disrupting applications that use it. We will make several software components multi- threaded and will provide a special multi-threaded external connector to binary storage. In Phase I we have already designed the changes to enable multi-threading in the necessary software components and we prototyped algorithms for the external connector to binary storage. We showed how our multi-threaded solution addresses requests for productive software development and eliminates performance bottlenecks. In Phase II we will: (1) Deliver multi-threaded access to complex and heterogeneous data stored in the scientific hierarchical file format as an external extension to the software, (2) Improve robustness of the software by contributing to the library, test suites and documentation, and (3) Set an example of systematic way of converting legacy software to thread-safety that we hope will trigger more community contributions to the software. By adding multi-threading to the software components, we are enabling development of other multi- threaded connectors for the software. This is an additional public benefit of our work. Discussions with customers have not identified multi-threaded solutions for the scientific and engineering user community that are general and scalable. The proposed solution is uniquely able to address community needs.