Creating Scalable Petascale File Systems using Application-Aware Network Offloading

Parallel ¿le systems are a key component of any high performance compute cluster used to perform cutting edge research, including many important DOE applications. As larger compute systems are developed and many applica­tions are becoming more data intensive, it is crucial that ¿le systems do not become the primary bottleneck. In order to meet the demand of such systems the meta-data server must be parallelized, and hardware of¿oad techniques can be utilized to improve performance and scalability of parallel ¿le systems. The meta-data server will be parallelized using low overhead RDMA operations and hardware based atomic con­sistency mechanisms. In addition, application-aware hardware of¿oad techniques, e.g., dynamic caching, hardware consistency protocol implementation, networking atomic operations, etc., can be used to improve existing parallel ¿le system implementations (e.g., Lustre). In order to demonstrate the feasibility of the proposed solutions, the bottlenecks of existing ¿le-systems (e.g., Lustre) were demonstrated using realistic experiments. Client-side metadata delegation was designed (based on network locking and a distributed caching architecture) to distribute the metadata workload and improve perfor­mance and scalability . Locking and distributed caching were demonstrated to perform much more ef¿ciently using of¿oading to network hardware. In Phase II, client-side delegation and associated SmartNIC of¿oad modules will be integrated with Lustre and LNET (Lustre¿s networking layer). To support this effort, SmartNIC network of¿oad modules will be created for TCP, RDMA, network atomics, and network collectives. In addition, as part of this project, support will be added to allow end-user application of¿oad development in order to broaden application support and allow support for a wider range of system layers (e.g., MPI of¿oad) and applications. Commercial Applications and other Benefits as described by the awardee: A scalable parallel ¿le systems will be required to build larger supercomputers and data centers required to perform cutting edge scienti¿c research and support large scale business applications. The techniques proposed would achieve this by leveraging the SmartNIC developed at RNET. The ¿le system extensions will be one of many SmartNIC applications used to accelerate many high performance computing solutions.