A Platform for Scientific Data Management, Modeling and Analysis with Machine Learning

While machine learning is showing great promise for discovering knowledge and patterns in data, it remains very hard to apply its techniques to the complex data which is at the core of DOE mission science, and which is prevalent in many industrial and commercial data science domains. Data complexity results in significant “dark data” that eludes analysis by researchers and business analysts, resulting in lost opportunities for discovery and the failure to fully leverage costly data collection efforts. Due to such data’s structural and/or processing complexity, machine learning remains difficult to integrate across multiple length and time scales, and across multi-domain science problems. The project will create and evaluate a novel platform for the flexible application of diverse machine learning methods and tools across a broad range of scientific disciplines as well as across diverse engineering, business, and public sector domains. The platform will enhance data management and analysis to make it easier to apply machine learning analytic methods across multi- scale/multi-physics problems that involve data of high volume, variety and velocity of growth and change. Expert domain specialists on the project will help to conduct proof-of-concepts evaluations in environmental science, computational chemistry, advanced materials and commercial exemplars. We seek here to deliver two key innovations: 1) creating workflow tools that enhance and simplify machine learning methods on complex data, and 2) enabling those workflows to quickly react and adapt to changes in the content and structure of datasets, models and workflow scripts. The project will make the resulting machine-learning-driven analysis platform available to biological, environmental and energy research communities across Department of Energy labs and facilities. In phase 2 we will further develop these capabilities with development and innovation in the systems, security, scaling and user interface areas. In Phase I, we prototyped innovations in the core dataflow model of machine learning computational processes, and in the management of data inputs, outputs and changes (the “evolution” of schema, data and programs). We evaluated the innovations on problems in four areas: chemistry, biochemistry (for Covid therapeutics), river ecology/hydrology systems, building energy modeling and prediction, and touched on tools for applying machine learning to data analysis problems in materials. In Phase II we will evaluate and guide the development of these machine learning workflow innovations by applying them to three problems in DOE mission science: environmental modeling of river biogeochemistry; chemical modeling of atomic and molecular force fields and free energy; and the discovery of advanced energy materials through multi-database analysis. We will also test the innovations on two domains representing promising commercial applications: precision agriculture and the industrial internet-of-things applied to building energy modeling and management. We will define a roadmap for addressing the broader engineering and business markets that can benefit from using structured machine learning workflow to analyze complex data. The project contributes back to the open source projects in workflow and machine learning from which we obtain our base software, providing benefits to the broad developer community.