Data Compression for Bandwidth Constrained Environments

We propose to extend existing free and open source software to construct a framework solution which implements a multi stage process to support the SBIR requirements. The baseline approach includes an encoding stage followed by a compression stage with optional encryption and transmission stages. Our approach referred to as"Framework for Rule-based Encoding and Stream COmpression (FRESCO)"leverages proven techniques developed for the finance industry to support high volume market data in a low latency manner. FRESCO's first stage analyzes source data and encodes it for the purpose of reducing redundant content and preparing it for optimal compression. Rules written in XML govern encoding behavior and are configuration managed to support data life-cycle. Rules are loaded and interpreted at runtime removing the need to modify encoding software. Automated tools analyze representative source data to assist in development of encoding rules. The second stage is responsible for receiving content from the encoding stage and selecting an appropriate best-of-breed compression algorithm optimized for the encoded data. FRESCO will accommodate pluggable compression components to achieve greater product flexibility. Preliminary testing with candidate data shows we are able to exceed the goal of 1100:1 compression.