Next-Generation Technology for the Extremely Efficient Storage, Distribution, and Processing of Nuclear Physics Data

Millions of dollars are spent every year in generating the tons of data that are necessary for addressing nuclear physics grand challenge problems at DOE facilities- Due to inherent technical and financial limitations at the “last- mile” of the data delivery pipeline, only a small fraction of these expensive Peta bytes of data is available for immediate access by end-users at any given time- This not only renders the users’ work inefficient, but also risks the waste of significant amounts of information that could very well contain the keys to answering grand challenge questions- This project aims at solving this issue by providing the ground-breaking data compression technology able to multiply the capacity of existing (and future) last-mile “live storage” facilities by factors larger than 4x and up to 9x without any additional investment in hardware- The technology, which will be packaged as an open-source set of tools, will be integrated with ROOT/IO and will provide the users with full control in the form of easy to understand/use/modify/adapt according to the specific needs and technical sophistication of each user- The data compression technology is based on a novel and very rich theory of application-driven losslesscompression that systematically analyzes the data in search of hidden redundancies that are not easily detected by traditional general-purpose compression- In addition, the theory provides fundamental mechanisms for achieving unprecedented lossless compression factors from a thorough understanding of the peculiar —and rich— interdependencies among the precisions of the data- This theory is complemented by very efficient and portable software implementations that we have developed in the last five years and which have already made paradigm- shifting impact of programs at NASA and DoD- Those implementations include the fastest compressors/decompressors and highest compression factors in the world- Already successful for network-data and in-memory compression, this project will realize the power of the proposed data compression technology for nuclear physics I/O and storage-