A Scalable High Performance Data Distribution Middleware for High Energy Physics Applications

Modern science and High-Energy Physics in particular, use scientific workflows to collect and analyze extremely large data sets using high-performance data acquisitions systems. As the requirements for scientific data processing systems continue to grow, fundamentally new approaches must be developed to meet future needs. The data distribution technology must support extreme data rates and must scale to 1000s of nodes. As the development of scientific workflows is highly collaborative, seamless evolution of algorithms and input/output data types must be supported without disrupting or complicating workflow development. The new solutions must be more robust and flexible than currently available solutions. Finally, as these systems grow larger and larger, effective software techniques are needed to control (startup, shutdown), configure, and manage scientific assets. This Phase II SBIR proposes a new approach based on the international OMG Data Distribution Service (DDS) standard to support scalable and extremely high-throughput messaging for High- Energy Physics applications. Our implementation, RTI Connext DDS, has been proven in high-performance industrial and military real-time distributed systems. DDS supports loose coupling, discoverable, structured data models, tunable quality-of-service, and extensible typesall of which are significant improvements over the prior generation of technologies and therefore, highly desirable in scientific data processing. Phase I research demonstrated that DDS is an attractive technology for scientific data processing. RTI Connext DDS exceeded the expected bandwidth goals while being very resource efficient. The developed communication patterns were shown to simplify programming and system management. Phase II research will productize the DDS features deemed essential for scientific data processing. Specifically, a high-throughput transport, a powerful data modeling capability, communication patterns, and a standards-based application programming interface will be productized. Feedback from the subject matter experts at national labs will be incorporated in the work plan to ensure alignment. The main objective of this research project is to meet the requirements of high-energy physics and Astrophysics applications. However, the resulting technology is applicable in a much broader context. For example, finance and healthcare domains often require systems that support extremely high throughput and low latency. Large-scale distributed systems in aerospace and defense, energy systems, factory automation, and transportation are critically dependent on having a robust data distribution middleware for physical infrastructure control and management. The technology developed under this SBIR will be immediately applicable in several other commercial as well as government programs. RTI is experienced in commercializing technology with 100s of successful customers and maintains its top 1% rank (a rating of 100 in the DoD commercialization achievement index) as assigned by the DoD.