Streaming Operational Analytics Platform

ATA’s proposed innovation, SOAP, is a platform for developing, deploying, monitoring, and maintaining machine learning-integrated streaming data pipelines. The solution will simplify the historically time-consuming, manual, and code heavy tasks associated with operational use of machine learning (ML) and streaming data management, including integrating new data sources, creating data pipelines, and applying ML. Leveraging a portable infrastructure and distributed processing and storage, SOAP offers the scalability needed to operate on the large volumes, velocities, and varieties of data typical of modern streaming data applications, including multi-sensor integration, digital engineering, and model and simulation. SOAP addresses the relative lack of investment into operationalizing streaming ML by directly providing a means for data engineers and scientists to create, monitor, and update ML-integrated data pipelines. SOAP includes two primary innovations: 1) services to execute ML models within lightweight, easily configurable streaming data pipelines, and 2) a service for monitoring deployed ML model performance for degrading accuracy, stability, and speed. We propose to develop the new services on top of mature, widely-adopted stream processing technologies while extending the functionality of more basic Machine Learning Operations (MLOps) tools. By promoting extensibility and ease of integration with outside data and services, this design is cost effective, leverages prior investment, and avoids the technical debt associated with single-tool data soloing. Approved for Public Release | 21-MDA-11013 (19 Nov 21)