Toward robust HDF5

A scientific binary data format and the I/O library have become the de-facto standard for managing large and complex data throughout the data lifecycle, from data acquisition to data archiving. While the format offers efficient storage and access to data, it comes with a major shortcoming: data may be lost if a file written according to the format by the I/O library becomes corrupted. Data corruption may occur when an application terminates due to internal error or system crash, or when an application is accidentally killed by a user. Another common data corruption scenario happens when a file is accessed in a way that is not supported by the I/O library, for example, two or more processes and/or threads modify the file at the same time. While in some cases the problem can be solved by re-running the application and regenerating the data, this may not be a solution for applications that run for a long time, or for data acquisition applications. When such a catastrophic event occurs, a binary file is often left in an inconsistent state due to missing pieces of internal metadata or corrupted metadata making it unreadable by the I/O library. With corrupted metadata it becomes impossible to recover any data stored in the binary file. Providing a solution for the data corruption problem is one of the long-awaited features by the users of the binary data format and the I/O library. The project will leverage existing architecture and functionality of the I/O library and implement coarse metadata journaling by saving snapshots of metadata on a filesystem of users’ choice. Those snapshots always represent a consistent state of metadata and can be used by a tool to restore the corrupted file. As a by-product of this work, the project will deliver an enhanced data streaming capability and a plugin mechanism for custom I/O drivers. In Phase I of the project, we will design and prototype the necessary changes to the I/O library and the data recovery tool. The software prototyped in Phase I and refined in Phase II will: Improve robustness of the I/O library and offer an operational mode that minimizes loss of data in event of application crash, Provide a recovery tool to restore corrupted data, Have an enhanced data streaming capability and a plugin mechanism for custom I/O drivers.