Storage Efficient Data Mining for High-Speed Data Streams

Sonalysts, Inc. and the University of Connecticut have developed a proof-of-concept prototype for a cross-disciplinary, efficient stream data mining and intelligent storage system that is portable and configurable for systems that ingest huge data streams and determine what data to store where. For the Phase II research, Sonalysts will build upon the prototype development to receive actual incoming data and metadata to store, add additional decision algorithms, execute multiple algorithms simultaneously against the same data stream, bundle groups of records into "events," support parallel alternative shunting algorithms and further utilize and generalize our metadata concept to support varying domains. Sonalysts will extend functionality to support configuration of trigger streams, the fixed length time step function, and weighting factors and their tolerances. Sonalysts will define COTS hardware and software to integrate with the storage decision system to support a physical prototype. We will continue to utilize Sonalysts’ commercial wXstation® software to demonstrate our implementation of these technologies. Best known for its ability to decode and display overlapping information such as international weather satellite imagery, wXstation ingests approximately 20 gigabytes of streaming data per day from multiple sources in various formats.