Scalable Real-Time Anomaly Detection for Rapid, Voluminous Data Streams in Large Scale Distributed Heterogeneous Computing Environments

To support global scale open sciences, the DoE HPC and exascale systems environments are relatively open and accept more risk in the interest of their primary mission of advancing science goals. However, scientific computing integrity and availability are paramount goals. Therefore, it becomes increasingly important to design data-driven cyber security solutions that effectively and efficiently derive actionable intelligence from heterogeneous sources of information using principled data analytic methods to defend against cyber threats and to ensure system integrity. In this phase I project, Kalos Technologies will research and develop a scalable distributed framework to collect and process extreme-scale scientific computing integrity data and knowledge from multiple sources that collectively represent the system under study, and develop and apply customized propriety real-time adaptive, streaming processing analytics in order to monitor, understand, maintain, and improve scientific computing integrity and computer & network security, combining software and hardware based approaches. Additionally, Kalos Technologies will develop means to learning and maintaining interdependent causal models of the scientific computation, exascale systems, and computer security in real-time to enable root cause analysis, better, faster recovery to reduce disruptions to scientists’ efforts. Early anomaly detection in streaming data can be extremely valuable in many domains, such as various government agencies, IT security (e.g., identify changes in employee behavior that signal a security breach), finance, vehicle tracking, health care, energy grid monitoring, e-commerce – essentially in any application where there are sensors that produce important data changing over time. The technologies developed offer significant improvements over existing methods of anomaly detection for streaming data in real time. In today’s world where the amount of data being collected is exploding, the opportunity for detecting anomalies is exploding along with it. The market demand for the developed technologies and the ensuing software products is enormous. The developed technologies are universally applicable in many different domains where real-time streaming data analytics are essential. Scientific computing integrity and availability are paramount for DoE HPC networking and systems. This project designs and studies data-driven cyber security solutions that effectively and efficiently derive actionable intelligence from heterogeneous sources of information using principled data analytic methods to defend against cyber threats and to ensure system integrity.