Semantic, Real-Time, Streaming Data Compression for MDA Modeling and Simulation Systems

M&S systems generate from 1GB to 250GB of data per simulation run that needs to be remotely transmitted, analyzed, and stored. Due to the limited bandwidth between geo-distributed elements, once a run has completed, data continues to be transmitted over the network up to 60 minutes afterwards; moreover, new runs cannot start until all of the data from the last run has been received. This downtime not only impacts the productivity of the 40-50 people that are involved in each run, but also the underutilized resources that are part of the system. The key goal of this topic is to be able to dramatically reduce this data transmission latency by applying targeted compression techniques, thereby allowing several more runs to be executed. We will develop an open, extensible, real-time, comprehensive and context-driven data compression framework building upon widely successful open source and open standards technologies. Our system will adaptively apply compression algorithms in real-time, supporting both heterogeneous and evolving streams of data. This will serve as a highly configurable and pluggable foundation for all M&S compression requirements. Approved for Public Release | 19-MDA-9932 (21 Feb 19)