An Efficient I/O Framework for In Situ Data Extracts

As computational capabilities advance, the disparity between computational power and I/O capabilities poses a bottleneck in high-performance computing when writing simulation data to disk for subsequent post-processing. In situ processing methods address this bottleneck by executing traditional post-processing procedures during simulation runtime, minimizing the need for extensive data writing, and accelerating time to insight. Through this approach, data extract artifacts can be created, storing reduced data to disk, preserving sufficient richness for subsequent post-processing tasks. Nonetheless, these data extracts often arise from small subsets of disparate processes, resulting in imbalanced loads that strain I/O resources and consequently diminish overall performance. We propose to expand development of an automated data aggregation system targeting wildly load imbalanced in situ data extracts. This will operate at scale, employing ADIOS, a state of the art I/O library, improving the I/O efficiency of extract-based in situ workflows. We will expand this effort to GPU I/O, to support modern HPC platforms and next generation solvers. The design will be validated with DOD CFD solvers which will be integrated with modern Catalyst to take advantage of the performance gains.