Accelerated in-situ analysis of multi-dimensional data

The massive volumes of multi-dimensional array-oriented data generated by NOAA programs and the scientific community at large are predominantly stored in industry standard Network Common Data Form (NetCDF). Key challenges exist in making use of data stored in netCDF: data sets are often too large to be copied and transferred across networks for every user, and each time data is accessed by an analytics tool it must be retrieved, subsets extracted, and subsequently formatted, among other requirements, which can account for 80-90% of the total time needed to insight. To unlock the enormous potential of petabyte scale netCDF-formatted data stored at different locations, in this SBIR Phase I project, AirMettle, Inc. with its research partners from the University of Wisconsin-Madison proposes to explore the feasibility of integrating in-situ analysis capabilities for multi-dimensional data (netCDF) into our highly innovative real-time smart data lake solution. Dramatically accelerated data analytics performed at the storage layer addresses key challenges noted within the Climate Adaption and Mitigation Topic. Reducing data traffic between sites, shrinking required compute resources, and lowering costs – all while accelerating climate analyses by an order of magnitude – would bring great benefit to NOAA and the broader scientific community.