QuAI - A Quality Assurance Infrastructure for Data-Centric Applications

Data processing is becoming an integral part of modern experiments and should operate with the timeliness and reliability appropriate to each project. There is a need for a customizable, resource-aware and dynamic mechanism allowing for automatic dissemination of the quality assurance data to the right parties at the right time. The Data Distribution Service (DDS) is a promising new technology that is expected to deliver these features but needs to be tested within and adapted for DOE applications. Tech-X therefore proposes to develop QuAI, a Quality Assurance Infrastructure for timely and reliable distribution of quality assurance data between concurrent data processing applications. The infrastructure will consist of reusable C++ library and Python objects facilitating use of DDS for the upcoming Joint Dark Energy Mission (JDEM) cosponsored by HEP/DOE and NASA. QuAI will allow data exchange over the wide area network with the security and Quality of Service adequate for the JDEM data processing needs. The project will be carried out in collaboration with the Ground Data System JDEM team at Fermi National Accelerator Laboratory. Software developed in Phase I consists of prototypes for the simplifying C++ library and Python wrappers for DDS entities and examples demonstrating the use of the developed software for distribution of the JDEM-relevant data (images and tables from FITS files and simple status data) and providing elements of the Quality of Service satisfying some JDEM requirements (getting data reliably and getting pre-published data). Commercial applications and other benefits: Software developed in this project will allow for quality-assured distributed data processing for DOE experiments and NASA missions. This project will bring Tech-X consulting business for developing custom extensions of QuAI, including custom security, for new HEP, NP and NASA teams, industrial control systems, and mission critical military applications. Adding Python support to DDSimplementations will result in strategic partnerships with DDS vendors.