Adding Streaming Capabilities to HDF5

Many applications involved in real-time data collection need to monitor their data while it is being collected. This need occurs in a large and growing number of applications that use common scientific hierarchical data formats for data management, such as aircraft flight testing, autonomous vehicles, particle accelerators, and many more. There are two shortcomings to these formats that make it difficult to use during data acquisition, and this can force applications into expensive and awkward alternate solutions: (1) the ability to write data at very high speeds and at the same time read back that data; and (2) the ability to recover lost data in the event of a system failure during data acquisition. This project will implement a scalable and crash-proof method that makes it possible, in real-time, to monitor data streamed at very high speeds and high volumes. Additionally, it will deliver tools to help users deploy the software and tune the software for their unique circumstances. During Phase I, a prototype was implemented that allows multiple processes to read data being written, and to do so within a predictable time interval. This implementation also lays the foundation to enable recovery of lost data if a system fails. Based on the findings and success of Phase I, we will develop a production version of the software. We will also develop production versions of convenience products to enhance and exploit the benefits of the new software, including a tool the recover data in the event of a system crash, tools to help users configure it so that it can perform correctly and optimally in their unique environment, and tools to help users understand and tune the software on their unique systems. A third set of Phase II tasks is aimed at implementations for alternative storage technologies, notably the Network File System and object stores. Commercial applications of this technology are found in nearly every area of science and industry with a need to monitor data during data acquisition. Benefits of this technology are that it improves the accuracy and productivity of applications, simplifies the software infrastructure needed to collect and monitor data at high speeds, gives users a cost-saving alternative to building parts of their data collection infrastructure, enables users to tune data collection systems to improve performance. This capability will benefit of a large variety of applications that collect data in real-time, including flight tests, autonomous vehicle control, electron microscopy, synchrotron operation, and genomics.