A Platform for Scientific Data Management, Modeling and Analysis with Machine Learning

While machine learning is showing great promise for discovering knowledge and patterns in data, it remains very hard to apply machine learning techniques to the complex data which is at the core of DOE mission science, and which is prevalent in many industrial and commercial data science domains. Data complexity results in significant “dark data” that eludes analysis by researchers and business analysts, resulting in lost opportunities for discovery and the failure to fully leverage costly data collection efforts. Due to data’s structural and/or processing complexity, machine learning remains difficult to integrate across multiple length and time scales, and across multi-domain science problems. The project will create and evaluate a novel platform for the flexible application of diverse machine learning methods and tools across a broad range of scientific disciplines as well as across diverse engineering, business, and public sector domains. The platform will enhance data management and analysis to make it easier to apply machine learning analytic methods across multi- scale/multi-physics problems that involve data of high volume, variety and velocity of growth and change. Expert domain specialists on the project will help to conduct proof-of-concepts evaluations in environmental science, computational chemistry, advanced materials and several commercial exemplars. We seek here to deliver two key innovations: 1) creating workflow tools that enhance and simplify machine learning methods on complex data, and 2) enabling those workflows to quickly react and adapt to changes in the content and structure of datasets, models and workflow scripts. The project will make the ML-driven analysis platform available to DOE BER and BES communities across DOE labs and facilities. In phase 1, we will achieve innovations in the core dataflow model of machine learning computational processes, and in the management of data inputs, outputs and changes the “evolution” of schema, data and programs). In phase 2 we will further develop these capabilities with development and innovation in the systems, security, scaling and user interface areas. We will evaluate and guide the development of these machine learning workflow innovations by applying them to three problems in DOE mission science: environmental modeling of river biogeochemistry; chemical modeling of atomic and molecular force fields and free energy; and the discovery of advanced energy materials through multi-database analysis. We will also test the innovations on two domains representing promising commercial applications: precision agriculture and the industrial internet-of-things applied to building energy modeling and management. We will define a roadmap for addressing the broader engineering and business markets that can benefit from using structured machine learning workflow to analyze complex data. The project contributes back to the open source projects in workflow and machine learning from which we obtain our base software, providing benefits to the broad developer community.