COMAND: Context-aware Machine Learning Middleware for Real-time Distributed Streaming Big Data Analysis

Emerging applications and instrumental systems produce massive streaming data every day. It is essential that the big streaming data to be quickly and accurately processed and analyzed to extract useful information for timely decision making. Despite substantial work has been proposed for streaming data processing and analysis, e.g., SAMOA, Jubatus, etc., many shortcomings have not well addressed. Statement of How this Problem or Situation is Being Addressed: InfoBeyond advocates Context-aware Machine Learning Middleware for Real-time Distributed Streaming Big Data Analysis (COMAND) to address the challenges of distributed massive streaming data analysis. COMAND includes machine learning (ML) based algorithms that is designed to facilitate its implementation and integration with the existing infrastructures and algorithms. The operation is implemented via a two-stage operational architecture for efficient and accurate processing of massive streaming data. We will first design and develop a middleware architecture that enables efficient and scalable capabilities in distributed environments. Next, we will analyze and develop three key algorithms, namely, Distributed Streaming Data Clustering (DISDC), Distributed Optimal Context-aware Data Classification (DOCDC), and Real-time Publish/Subscribe Data Model Update (RPSDMU) for the middleware. The system is wrapped by APIs and Adapters for facilitating the implementation and integration of COMAND with the existing systems. Commercial Applications and Other Benefits: Once developed as COTS/GOTS products, COMAND will significantly advance distributed streaming data analysis capabilities for organizations and scientific community. • Enabling Real-time Massive Streaming Data Analysis: Several business applications will be beneficial from the proposed COMAND system including cloud-based services and applications, social networks, e- commerce and advertisements, sensor networks, surveillance systems, etc. • Enhancing Cyber-security and Preventing Cyber-attacks: Additionally, the proposed framework COMAND can also be applied to network security applications, where several classifiers collaborate, learn, and detect the abnormal behavior of network traffic in a real-time distributed manner to prevent malware or security attacks. It can also be used for criminal services where useful information for tracking criminals can be extracted from multiple information sources in the Internet. • Enabling Cost-effective Commercial Grade Big Data Processing: COMAND is flexible for real-time distributed streaming data processing and analysis with heterogeneous computing resources. With the built- in APIs and Adapters, COMAND allows it to be quickly adopted and transitioned to the emerging Big Data processing markets. COMAND could be first transferred into our commercialized NXdrive cloud storage system for analyzing massive streaming Big Data, and then transitioned to our established Big Dag customers including EMC, Terada, HP, etc.