High-Speed, Large-Scale, Extensible, SCADA and Integration Platform for Machine Learning and AI Data Collection

A major barrier to advances of machine learning (ML) and artificial intelligence (AI) in manufacturing and largescale DOE facilities is incomplete and inaccurate data sets. The proposed distributed framework offers a versatile, high-performance, and extremely scalable data consolidation platform, adept at handling vast amounts of data. The "data mesh" or "distributed data lake" architecture allows domain data owners to manage their specialized data and treats data as a product. This approach empowers data owners to gather and disseminate data without a centralized team, resulting in a more comprehensive and well-documented dataset. Instead of replicating data into a central repository, the architecture integrates or "wraps" current data sources, incorporating unstructured data like images, waveforms, video, and text. This approach to decentralized data management is made possible by a no-code/low-code application development platform. GraphQL is used in the data mesh to provide a consistent and reliable method for users to access data. Despite being a distributed system, clients can access data as if it were a single, unified structure. This GraphQL "federation" functionality simplifies the development and maintenance of client applications while reducing potential errors from inconsistent data access patterns. This helps the data consumers, such as data scientists, to find and utilize data for diverse applications. Distributing and federating data in this manner can improve performance by enabling complex queries to span multiple data sources. Each GraphQL service resolves its part of the query, minimizing unnecessary data transfer, resulting in faster response times and reduced bandwidth usage. In Phase I, support for GraphQL was added to the no-code/low code application builder to expose data from any single node. Phase II research will continue this work, focusing on improving the usability of the application builder. Phase II will also expand testing and installation in a high energy physics accelerator facility and demonstration of this architecture for machine learning. Phase II will continue improving the no-code/low-code application development environment. This resource empowers data domain owners to gather and distribute data without relying on conventional software development skills. Additionally, the integration of high-energy physics instruments will be further developed, bolstering our support for the EPICS data collection and control protocol utilized in research facilities. This technology can be used in manufacturing and other industries where structured and unstructured data can be effectively used in machine learning to optimize processes, increase productivity, and improve quality of the products.