Real-time Alpha-Flow Traffic Management System

As supercomputing speeds increase and storage costs drop, scientists running applications in various disci- plines, such as high-energy physics, genomics, climate studies, etc., generate datasets of ever-increasing sizes at distributed locations around the world. Backbone Research-and-Education Network (REN) providers such as DOE’s Energy Sciences Network (ESnet) and Internet2, provide high-speed, reliable network services to support these scientists located at various US Department of Energy (DOE)’s national laboratories and universities. Our work will identify, in real-time, very large data flows across these networks, moving them to alternative network paths or queues so that interactive applications such as voice, video, and applications can retain a high quality of service thereby extending the interval to the next inevitable costly equipment upgrade cycle. We are developing a system called real-time Hybrid Network Traffic Engineering System (rHNTES) which will tap the traffic between networking installations to determine which network flows are large, ongoing, and of high bandwidth; i.e. elephant flows. We consider the timely completion of these elephant flows less important than interactive network traffic such as voice, video or applications and as such, change the network priority such that elephant flows are lower priority than other more import network traffic. Our elephant flow identification is accomplished in real-time, using off-the-shelf network processors and innovative software algorithms. Once identified, we will reprogram the switching infrastructure to ensure these flows do not impact the higher priority flows. This project will develop a software packet processing engine using readily available commodity network processors which will be able to analyze network packets at up to 100 billion bits per second. We will use this software engine to develop our real-time HNTES (rHNTES) system based on prior HNTES algorithm work. The network routers will be reprogrammed on-the-fly to mitigate the impact of each of the elephant flows we identify. This system will then be integrated into existing network performance monitoring tools to help identify where network bottles occur. Commercial Applications and Other Benefits: This work will enhance collaborative science making more efficient use of existing and future networking technologies that will help usher in scientific breakthroughs.