Protecting the confidentiality and integrity of data stored in HDF5

Data has become one of the most valuable assets. Protecting data from unauthorized access is one of the highest priorities for all organizations that collect, store and process sensitive data, including government, financial, medical, pharmaceutical companies, and research institutions. Many commercial and open-source data management solutions offer multiple security mechanisms including encryption for data at rest and data in transit. Unfortunately, similar data protection is not available in a scientific binary data format that has become a de facto standard for managing large and complex data and is heavily used by government, academia and industry. The binary format is not suited for storing sensitive data due to its open specification and no built-in mechanism to protect data in the corresponding open source I/O library. The content of a file can be viewed with open source and commercial tools or simple programs written in high-level languages, e.g., Python. Open file format and easy access to data creates a risk to storing sensitive data in the format and presents a barrier for its wider adoption by industry, especially by bio-medical and pharma industries. Encryption of all data in the binary file and ability to provide random access to it is one of the long- awaited features by the users of the binary file format and I/O library. The project will implement data encryption in the binary file while allowing random access to all data without losing functionality of the I/O library and without changes to the file format itself. Access to encrypted data will be transparent to the applications and will not require a special coding effort. We will leverage the existing architecture of the I/O library to implement the encryption feature while enhancing some of the existing software components and documentation. In Phase I of the project, we will design and prototype encryption in the binary file format. We will also prototype the required enhancements to the I/O library. The software prototyped in Phase I and refined in Phase II will: Prevent unauthorized access to data on storage. All data and metadata stored in the binary file will be protected and will be visible only to applications that have authorized access without a need to decrypt the whole file and regardless where the file is stored. Protect data when moved between different storage types and systems (e.g., between host computer and Cloud storage); when accessed over the network; when accessed directly in the Cloud or on a remote server.