Implementation of an Energy-Saving Bro-Aware Load Balancer at 100 Gbps with Closed-Loop Flow Policy Control

In an increasingly hostile computing environment, Network Intrusion Detection Systems (NIDS) serve an indispensable role in preserving the integrity of computer networks. This comes to manifest as the Department of Energy (DOE) is working at a national level to secure a number of strategic network entry points using Bro, a powerful NIDS developed by the networking group at the International Computer Science Institute (ICSI) in Berkeley, California. Existing NIDS such as Bro, however, have been for the most part deployed as single-node appliances protecting a specific network asset. This architecture is being driven to a breaking point by two independent realities: first, as network data-rates increase, single-node NIDS boxes are being overwhelmed by the quantity of computation they must perform to continuously secure the network; second, as attacks become ever more sophisticated, NIDS have to incorporate more complex traffic analysis heuristics that further stress the systems processing capacity. To address these trends, ICSI has extended the functionality of the Bro NIDS with a cluster architecture that allows for the logical aggregation of multiple Bro nodes. The cluster architecture provides a way to arbitrarily scale the performance of the NIDS with the exception of one element: the front-end load balancer. In a cluster solution, the load balancer is the performance bottleneck because it is the only element that must process all the traffic as an indivisible trunk. To address this system bottleneck, we propose to design and implement a high-performance energy-saving load balancer that can distribute traffic at line rates of 100 Gbps. We argue that due to the strong heavy-tailed nature of network traffic which shows that most of the relevant information from a traffic analysis perspective resides in a small portion of the total traffic an optimal design will tend to shift intelligence and bring Bro-awareness toward the front-end load balancer to offload traffic from the back-end nodes. We show that by doing so, energy consumption in the cluster can be reduced by a factor of 2X to 10X. In Phase I, we built a scaled-down version of our load balancing solution at speeds up to 10 Gbps, capable of load balancing traffic and of offloading irrelevant traffic using dynamic feedback from the back-end cluster. In Phase II, we plan to scale our current solution to support speeds of 100 Gbps and use our intelligent load balancing algorithm to offload traffic from the cluster, with the objective to reduce energy consumption by a factor of 2X to 10X. The resulting technology will have commercial applications in the field of cloud computing, where large datacenters are in need to move massive amounts of data in a secure manner. Such datacenters can be found in both government facilities such as the DOE and commercial facilities such as Internet service providers and cloud computing providers.