MIO: Tailoring Multi-level Decomposition for Storage I/O for Computational Applications

As computational sciences usher in a new era with vastly improved model fidelity empowered by the recent advances in high-performance computing (HPC), there is an urgent need to re-design the data analytics toolchain in the commercial market so that it can better adapt to the rapid growth of data coming out of large-scale simulations and maintain a high level of productivity. To address this need, this project proposes to develop a multi-level data retrieval library for hierarchical storage systems, so that users can request the minimum amount of data from fast storage tiers and pay higher input/output overheads only when needed. This project aims to address Focus Area 13 Information Technologies for Science Data to provide S17.01: Technologies for Large-Scale Numerical Simulation. Through co-designing the decomposition and recomposition over hierarchical storage, this project will offer new capabilities to the user communities, including those in the federal market (e.g., GEOS-5, FUN3D developers) and private sectors, and vastly improve the efficiency and flexibility of data analysis.