Creating Scalable Petascale File Systems using Application-Aware Network Offloading

Many DOE applications are becoming more data intensive, and petascale computing systems are being developed to respond to this need. As these larger computing machines are developed, it is crucial that their file systems do not become the primary bottleneck. In order to meet the demand of such systems, the metadata server must be parallelized, and hardware offload techniques must be utilized to improve performance and scalability of parallel file systems. In this project, the meta-data server will be parallelized using low overhead RDMA operations and hardware based atomic consistency mechanisms. In addition, hardware offload techniques e.g., dynamic caching, hardware consistency protocol implementation, and networking atomic will be used to improve existing parallel file system implementations e.g., Lustre. Commercial Applications and other Benefits as described by the awardee: The scalable parallel file systems developed in this project will support the construction of larger supercomputers and data centers, required for cutting edge scientific research and large scale business applications.