Flow Ordering and Hierarchical Bottleneck Identification for High Speed Data Networks

Driven by new bandwidth-intensive applications such as big data, cloud applications or Internet of Things, data growth is exploding worldwide as it continues to expand into all areas of society including business, science or leisure. Key to ensure the productivity of our economic and social systems is the transportation of these ever increasing large datasets in a timely and cost-effective manner. For instance, backbone Research and Education Network providers deliver high speed, reliable network services to scientists located at national laboratories and universities across the nation and abroad. While technologies such as Software Defined Networking have been key in providing the degree of programmability to enable these new advanced network architectures, their true potential performance is being stifled by an existing inability to measure the state of the network with precision. For instance, on the demand side, upon a network congestion collapse situation, how can network operators quickly identify the flows that are responsible and what will be the effect of a correcting policy to the overall performance of the network? Similarly, on the supply side, where are the bottleneck links of a network, in what hierarchical structure are they organized, and how will a flow policy modify such bottleneck structure and propagate through the network? In this project, we will carry out a technical work plan to develop a new network analytical framework and technology to address the lack of fine grained visibility and understanding of flows and bottlenecks in the space of high speed SDN networks. The proposed technology is based on a new mathematical framework that provides new scalable, high performance algorithms to characterize the inherent structure and hierarchical relationships of flows and bottleneck links. The objective of the Phase I is to develop a prototype software optimization library and engine which will implement the high performance algorithms to compute both the flow gradient and the bottleneck precedence graphs of a network as well as the core network optimization algorithms developed during this project. The proposed technology will lead to comprehensive flow and bottleneck visibility of advanced high speed networks towards higher system utilization and more reliable performance. These communication systems are the tools used by scientists and engineers across the nation in National Research and Education Networks to execute compute-intensive science applications that are tightly connected with real societal problems in areas including business, utility, financial, education, and critical national infrastructure systems. The technology will also lead to commercialization into Internet Service Providers, and large scale data centers such as those built by the largest cloud providers leading to better productivity for users, public and private organizations and enterprises that rely on high speed data network communications.