Framework for Large Streaming Data Analysis

This Small Business Innovation Research Phase II project will develop a scalable system capable of ingesting data from disparate media sources, application of custom workflow and mediation processes, and serving media to multiple, concurrent subscribers. The proposed system will feature a plug-in data framework combined with a public application programming interface (API) enabling first-party and third-party developers to create custom modules that seamlessly integrate new media formats and workflow algorithms without any changes to the underlying architecture. The key innovations in this effort include: i) a service-oriented architecture (SOA) using standards compliant command and control protocols allow for future system expansion, ii) automated resource management capabilities that distribute processing requests across redundant nodes running on remote static and/or cloud based hardware, iii) modular stream handling capabilities enabling muxing and demuxing of multiple data streams at any point in the processing pipeline, iv) plug-in frameworks and templating schemes allowing integration of data types, analysis components, and workflow processes post-deployment, and v) intuitive user interfaces enabling simplifying system management tasks and enabling debugging of complex analysis processes. The Phase II effort will produce a prototype system and reference implementation of required plug-ins to demonstrate these capabilities.