High-Speed, Large-Scale, Extensible, SCADA and Integration Platform for Machine Learning and AI Data Collection

The US Department of Energy’s (DOE’s) scientific user facilities provide access to the world’s most advanced research instruments and produce increasingly larger quantities of data. DOE’s Basic Energy Sciences (BES) scientific user facility instruments for x-ray, neutron, and nanoscale science are among the world’s most productive, serving over 16,000 users per year with impact reported in nearly 7,000 publications and resulting in unprecedented quantities of scientific data. While this record is impressive, using the rapidly growing data stream to reach its full potential will require new innovations to solve a variety of technical challenges in data acquisition, control, modeling, and analysis. Artificial intelligence and machine learning (AI/ML) have opened corresponding new avenues in optimization, efficient surrogate models, data analytics, and inverse problems. These intriguing capabilities suggest that AI/ML can greatly accelerate the quest to probe and understand fundamental phenomena across a vast range of length, time and energy scales, potentially leading to transformative advances across scientific disciplines. The challenge of AI/ML is data. Applying AL/ML algorithms is the end of a very intensive chain of data collection and manipulation. Data collected from instruments and other sources in scientific facilities range from real-time to parsing log files and from scalar time-series data to images, waveforms and other vectors in unlimited dimensions. For large-scale facilities, the data problem is magnified by the physical scale, the volume of data and the data speed, with the potential of sub-microsecond time-series data. Large scale facilities crucially require AI/ML tools throughout the lifetime of an experiment: not just for data analysis, but also for data creation, acquisition, and storage. This proposal addresses the challenges of collecting data from instruments in a km-scale facility particularly the challenges of time-synchronization of computers to allow accurate timestamps of data collected at high speed. The challenges of managing this disparate data, ranging from simple data point, to waveforms to images, and allowing the an AI/ML model to query these in a uniform fashion are addressed with a novel solution of using a combination of commercial and open-source software to build consistent schema and data organization.