02aToward multi-threaded concurrency in HDF5

Standard scientific software and scientific hierarchical data formats for data management are widely used in large-scale simulations of physical phenomena, aircraft flight testing, high-speed neutron and X- ray detectors data acquisition, and other applications. One of the known limitations of the existing data management software is a lack of multi-threaded support for accessing data. It creates performance, deployment, and adoption barriers for multi-threaded applications, especially for machine learning applications. To date, no effort has been made to overcome this restriction, mostly due to the difficulty of retrofitting a large code base with thread concurrency. Recent developments in the architecture of one of the most used data management software allows us to suggest a strategy of retrofitting thread-safety incrementally, without affecting any other development going on in the software, and without disrupting applications that use it. We will retrofit a few components of the software and provide a special external connector to binary storage that will allow applications to get advantages of multi-threaded access to data in the relatively short period of time of Phase I and Phase II of this proposal. In Phase I we will prototype retrofitting of thread-safety into necessary software components along with designing and prototyping an external connector to binary storage. We will show how our solution addresses requests for productive software development and eliminates performance bottlenecks. The software prototyped in Phase I and refined in Phase II will: (1) Deliver multi-thread access to complex and heterogeneous data stored in a scientific hierarchical file format that is a de facto standard for scientific and engineering applications in government, academia and industry; (2) Improve robustness of the open-source software by our contributions to the library, test suites and documentation; and (3) Demonstrate a systematic method of converting legacy software to thread-safety that we hope will trigger more community contributions to the open-source project. Discussions with customers have not identified multi-thread solutions for the scientific and engineering user community that are general and scalable. The proposed solution is uniquely able to address community needs.