Real-time, Automated Decision Making and Analysis Using Large-Scale Data Streams

We consider large, heterogenous data streams comprised of multiple components that may lack underlying models. The problem is to reliably and automatically decompose these high-dimensional objects into a 'background' of simple patterns and a smaller number of idiosyncratic 'foreground' events that represent deviations from the background trends in the presence of unknown amounts of noise. This is a complex problem, made more difficult by the existence of severe computational and storage constraints, relative to the size of the data that must be processed. We propose a computational framework that is able to robustly solve this problem in an efficient and highly-scalable manner. Early results from Phase I demonstrate our algorithms substantially outperform current state-of-the-art along multiple metrics on multiple real-world data sources. We propose to build upon this promising start by scaling our data processing platform to handle substantial increases in data volume, velocity, and variety and integrate it for use in real-world deployments.