Correlating and Analyzing Network Data through Interpretable Decompositions (CANDID) (1a)

Having operational awareness and visibility into networks has a critical effect on the national prosperity and security. It is very important to allow DOE scientists to operate uninterrupted on high-speed networks and enable them to make scientific discoveries. It is crucial to manage DOE, DOD, and commercial networks for enabling smooth and secured operations. Deep network visibility requires advanced correlation and analysis of different types of data collected from the network. Existing network analysis tools lack the sophistication and/or scalability to provide deep network operational visibility. How the Problem is Being Addressed CANDID will address the network visibility needs of network analysts through application of advanced tensor-based analytics and building an usable and scalable network analysis tool. CANDID will take in network metadata, transform the data into “tensors,” use a high-performance tensor-based analysis engine to analyze network tensors, interpret and identify actionable network patterns from the output of the analysis engine, and display a well-coordinated picture of the different activities and events in the network traffic. Phase I Project In Phase I, we will develop a prototype of a scalable network analysis tool that will build upon a mathematically sophisticated tensor analytics engine and be usable by network analysts in their existing environment. Towards this goal, we will develop capabilities to (1) transform network flow data into a suite of diverse tensors tailored to provide deep network traffic analysis, (2) quantify the significance and reproducibility of the results of tensor analysis, and (3) present clear and actionable network patterns to the user. We will demonstrate the ability to provide a deep post-hoc analysis of network flow data. Commercial Applications and Other Benefits A commercial product developed from the mature implementation of network analysis technology developed through this effort will empower the network analysts at DOE, DOD, and commercial enterprises. The product will help to accelerate innovation and improve security. Hence the product will have huge social and economic benefits, and contribute to national and global prosperity and security. In addition to DOE, DOD and other government agencies, this product will also be significant for commercial sectors such as finance and health.