A Cloud-based System for Complex Real-time Data Integration, Analysis, and Learning (CoRDIAL)

Rapid increase in observational data, brought about by the development and availability of Internet-of-Thing devices and measurement technological advancements, is posing a Big Data challenge for many domains of science and engineering, both research and commercial. These types of Big Data, however, are fundamentally different from the Big Data that exist in the business intelligence community although they share some of the same characteristics of high volume, velocity, and variety. Timely extraction of useful knowledge from these data will require different forms of information processing, storage and retrieval, as well as data analytics and predictive analytics. Failure to successfully adapt to the new situation may cause such data to never get analyzed, leading to the so-called “Dark Data” problem. This project aims to develop a cloud-based Complex Real-time Data Integration, Analysis, and Learning (CoRDIAL) system to address many of the aforementioned complex data challenges listed under Topic 1. Specifically, Phase I of this SBIR research aims to significantly streamline the access to Big Scientific Data streams by providing an automated and on-demand data management infrastructure, a unified approach for organizing and provisioning complex scientific and engineering data, a common platform for data analysis and visualization, and a set of machine-learning-driven predictive analytics tools for analyzing the complex data. The prototype system will be implemented in the cloud computing environment using complex subsurface data sets to demonstrate feasibility and essential system features.For Phase II, the prototype system will be expanded into a full turnkey solution that can be deployed in a cloud computing environment with minimal user involvement. This feature will significantly lower the barrier to a much needed platform where scientists and engineers can untangle the values hidden in Big Scientific Datasets, expediting the knowledge discovery. To maximize the economic benefit of such a system in the cloud environment, system’s components will be reexamined for implementation using cloud serverless architecture or similar type of resources.