Flow Ordering and Hierarchical Bottleneck Identification for High Speed Data Networks

Driven by new bandwidth-intensive applications such as big data, cloud applications or Internet of Things, data growth is exploding worldwide as it continues to expand into all areas of society including business, science or leisure. Key to ensure the productivity of our economic and social systems is the transportation of these ever-increasing large datasets in a timely and cost-effective manner. For instance, backbone Research and Education Network providers deliver high speed, reliable network services to scientists located at national laboratories and universities across the nation and abroad. While technologies such as Software Defined Networking have been key in providing the degree of programmability to enable these new advanced network architectures, their true potential performance is being stifled by an existing inability to measure the state of the network with precision. For instance, on the demand side, upon a network congestion collapse situation, how can network operators quickly identify the flows that are responsible and what will be the effect of a correcting policy to the overall performance of the network? Similarly, on the supply side, where are the bottleneck links of a network, in what hierarchical structure are they organized, and how will a flow policy modify such bottleneck structure and propagate through the network? The objective of the project is to implement GradientGraph (G2), a network optimization platform that will enable operators to perform advanced traffic engineering and capacity planning on high-speed data networks. During the project, GradientGraph will be integrated into a variety of networks, including Research and Education networks, Supercomputers, large-scale data centers, cloud and carrier networks. During Phase I, Reservoir Labs (1) developed a prototype of G2, (2) filed one patent, (3) published three scientific papers, and (4) demonstrated its capabilities to key industry players. As a result, the Phase II project received five letters of endorsement from some of the top Research and Education networks institutions and networking vendors. The proposed technology will lead to comprehensive flow and bottleneck visibility of advanced high-speed networks towards higher system utilization and more reliable performance. These communication systems are the tools used by scientists and engineers across the nation in Research and Education Networks to execute compute-intensive science applications that are tightly connected with real societal problems in areas including business, utility, financial, education, and critical national infrastructure systems. The technology will also lead to commercialization into Internet Service Providers, and large-scale data centers such as those built by the largest cloud providers leading to better productivity for users, public and private organizations and enterprises that rely on high speed data network communications.