From data to models and analytics: An integrated scientific computing platform to accelerate environmental research

Advances in sensor-based field observations, multi-scale models, and computational technologies in the Environmental Systems Science (ESS) provide new opportunities for applied research and development in academia and industry- These same advances are producing heterogeneous and disparate data of growing size, and increasingly varying types and formats- As a result, analysis now requires multiple components to be stitched together into complex workflows, before they can be analyzed and used for modeling- However, currently, most workflows for data-acquisition, management, analysis and integration into models are fragmented and specialized for one-off efforts- To reduce this fragmentation, there is a need for a unifying platform that enables data management, analytics, and collaborative research supporting efficient data post-processing and cross-domain modeling across a wide variety of data types and projects- Furthermore, there is an urgent need to rethink traditional workflows which involves moving data between resources (for e-g-, between compute and visualization system) as needed for visualization and hypothesis formulation- This is especially relevant for HPC based workflows because computing costs (both HPC and cloud) are becoming cheaper, while data transfer, be it in memory or external storage, is becoming comparatively more expensive- What is needed is the ability to compose elaborate yet flexible HPC workflows, reflecting current operational costs- These flexible workflows need to support analysis preparation, simulation, and analysis/visualization of results, as well as situations supporting adaptive analysis and even optimization- To address challenges described in the previous section and to advance high performance and throughput computing via flexible workflows in the ESS domain, Kitware and its collaborators will develop an integrated open-source platform enabling flexible workflows for model-data integration, in-situ analyses, and visualization- This platform will provide unified data interfaces for heterogeneous data sources and types with an interactive front-end for exploratory analyses- The design of this platform will ensure modular design, loose coupling, and seamless integration across components to guarantee maximum flexibility and existing code-reuse- This integrated scientific platform will use Kitware developed VTK and Girder, and community-led Jupyter effort in a loosely coupled modular design-