An Efficient I/O Framework for In Situ Data Extracts

Computational power has far outgrown modern I/O capabilities, leading to a bottleneck in high performance computing when attempting to write generated data to disk for future post-processing. In situ processing techniques mitigate this bottleneck by performing classically post-processing procedures while the simulation is running, without writing significant data to disk. Data extract artifacts can be generated in this way, storing reduced data to disk, still rich enough for further post-processing. However, data extracts are frequently generated with data on small subsets of processes, leading to imbalanced loading, straining I/O and reducing overall performance.  We propose the development of an automated data aggregation system targeting wildly load imbalanced in situ data extracts. This system will operate at scale, improving the I/O efficiency of extract-based in situ workflows. The design will be test driven from the beginning, focusing on quantitative performance metrics for future general and machine specific optimizations. Multiple in situ platforms will be targeted, including ParaView Catalyst and VisIt LibSim, ensuring wide availability of the data extract aggregation capabilities.