Protecting the Confidentiality and Integrity of Data Stored in HDF5.

Data has become one of the most valuable assets. Protecting data from unauthorized access is a top priority for all organizations that collect, store, and process sensitive information, including government agencies, financial institutions, medical organizations, pharmaceutical companies, and research institutions. Many commercial and open-source data management solutions offer multiple security mechanisms, including encryption for both data at rest and data in transit. Unfortunately, similar data protection mechanisms are not available in the scientific binary data format, which has become a de facto standard for managing large and complex datasets. This format is heavily used by government, academia, and industry. The binary format is not suited for storing sensitive data due to its open specification and lack of built-in mechanisms to protect data within the corresponding open-source I/O library. The content of a file can be accessed using open-source and commercial tools, or through simple programs written in high-level languages (e.g., Python). The open file format and ease of access to data create a significant risk when storing sensitive information, presenting a barrier to its wider adoption in industries like biomedical and pharmaceuticals. Encryption of all data within the binary file, while allowing random access to that data, is one of the most requested features by users of the binary file format and the open-source I/O library. The project will implement data encryption within the binary file while maintaining the ability for random access to all data stored in the file. This will be achieved without compromising the functionality of the open-source I/O library or requiring changes to the binary file format itself. Access to the encrypted data will be transparent to applications and will not require any special coding effort. We will leverage the existing architecture of the I/O library to implement the encryption feature, enhancing some of the existing software components and documentation. In Phase I of the project, we designed and prototyped encryption for the binary file format using low-level I/O drivers. We also prototyped the necessary enhancements to the sequential I/O library. In Phase II, we will finalize and release our solution as a plugin to the off-the-shelf open-source sequential and parallel I/O libraries. The encryption feature will: Prevent unauthorized access to data on storage: All data and metadata stored in the binary file will be protected and visible only to applications that have authorized access, without the need to decrypt the entire file, regardless of where the file is stored. Protect data when moved between different storage types and systems (e.g., between host computers and cloud storage), when accessed over the network, or when accessed directly in the cloud or on a remote server.