AnomLoc: A perfSONAR-based Distributed Network Anomaly Detection and Localization

Data-intensive scientific applications incline to high performance computing which is getting more and more widespread in supercomputing centers, research laboratories, and universities. DoE and many organizations need an automatic and adaptive network analysis tool for effective anomaly detection and localization in the high speed network. Currently approaches, Pythia, APD, etc., are unable to provide such a function effectively, accurately, and user-friendly. Statement of How this Problem is Being Addressed: InfoBeyond advocates a perfSONAR-based distributed network anomaly detection and localization scheme (AnomLoc) to address the challenges of performance problem diagnosis in distributed data-intensive-oriented networks by relying on perfSONAR legacy measurement infrastructure. AnomLoc includes Q-statistics and convex-optimization-based algorithms that are designed as pluggable tools to facilitate its implementation and integration with the existing infrastructures and algorithms. It is to be developed as a pluggable tool for real-time distributed anomaly detection and localization. In the first step, we perform data acquisition and purging on perfSONAR measurement data to filter out the misleading data. In the second step, we carry out anomaly detection and localization on the pruned data by relying on four key algorithms, namely, Sparse Principal Component Analysis (SPCA), Graph-based SPCA (GSPCA), Karhunen-Loeve-based SPCA (KLSPCA), and Graph-based KLSPCA (GKLSPCA) for anomaly localization. Additionally, we also develop APIs for the AnomLoc modules for easy implementation and smooth integration with the existing perfSONAR infrastructure. Commercialization Application and Other Benefits: AnomLoc targets to provide real-time distributed anomaly detection and localization in data-intensive-oriented networks while offering extendibility and flexibility to the users and developers. Once developed as products, it significantly advances distributed network performance problem diagnosis capabilities for organizations, enterprises, and scientific community: DoE, Government, and R&E Network: AnomLoc can be applied to DoE, government, and R&E network for detecting and localizing the abnormal behaviors of the network traffic in a real-time and distributed manner. The R&E network consists of thousands of national and international research and education communities. It provides a globally collaborative environment allowing researchers to collaborate and work together wherever they are located. In U.S., the R&E network comprises ESnet, Internet2, universities and federal labs. It also includes GEANT in Europe, AfricaConnect in Africa, TEIN in Asia-Pacific, and RedCLARA in Latin America. AnomLoc enables new security protections via real-time high-speed network diagnosis. Commercial Network Performance Monitoring: The enterprises and organizations are the prime target of AnomLoc market. This includes long-term target customers such as commercial Internet providers (e.g., AT&T, Verizon, Sprint, Time Warner Cable, Comcast, etc.), and large cloud providers (e.g., Amazon, Microsoft, Box, Google, YAHOO, etc.). Open Source Projects and Toolkits: In business model transition, AnomLoc can be integrated into popular open source projects (e.g., Nagios, Smokeping) to enhance its visibility and viability. In addition, AnomLoc modules can be incorporated into Linux distribution such as RedHat along with Apache Foundation Kit as part of their standard package repository for network performance monitoring. This open source integration will help promote AnomLoc and expedite its dissemination.