Novel Distributed Processing Environments

DoD data processing requirements are dramatically evolving. Dataset size and variety are rapidly increasing, as are the speed at which the data is generated and the need for faster response times. However, the problem is more than simply having to deal with data size or speed of arrival. Rapidly evolving problem spaces and requirements mean that the processing and analysis solutions also must be quicker to develop and more agile. This proposal introduces a high-speed, cloud-native analysis framework based on a standard set of analytical components, which is capable of addressing a wide variety of data shaping, discovery, and situational awareness problems. Our technical approach applies a hybrid hierarchical reasoning framework via a standard set of analytical elements to extend a stream-based execution platform enabling the rapid production and evolution of solutions to complex analysis problems. A key aspect of the proposed technology is tailoring the individual analysis components for high throughput as well as horizontal scalability. The functional depth is built through the aggregation of many atomic processing components as opposed to fewer more complex modules and the stream-based runtime environment will support on-the-fly changes to live operational analysis networks to adapt rapidly to evolving problem spaces.