H-WFQ for Isolation through Resource Allocation in HPC Storage

High-performance computing (HPC) systems at the Department of Energy (DOE) and other research institutions are critical for large-scale scientific discovery. These inherently multi-tenant facilities, with millions of processes active simultaneously, necessitate robust mechanisms to ensure high performance and stringent security across multiple users. However, existing resource allocation methods often struggle to dynamically and fairly distribute system resources under heavy load, leading to performance bottlenecks and potential security vulnerabilities. To address these challenges, we propose integrating Hierarchical Weighted Fair Queueing (H-WFQ) into our existing analytic data storage system. By implementing H-WFQ, we will enhance user isolation by controlling and limiting resource access at both tenant and process levels, thereby significantly improving security and performance. During Phase I, we will develop methods to integrate H-WFQ support into our analytic storage system to strengthen user isolation. Specifically, we will design multi-level resource management mechanisms for dynamic, fair resource allocation among tenants and users. We will validate the enhanced system's effectiveness through performance and security testing in simulated HPC environments and ensure compatibility with the Globus data management services. This integration will significantly enhance the efficiency, security, and performance of HPC systems by providing robust isolation aligned with administrative policies. In Phase II, we aim to commercialize this solution for large multi-tenant HPC facilities. Strengthening user isolation protects sensitive data and ensures compliance with regulatory standards across industries, representing a breakthrough in resource management for HPC systems critical to scientific and technological advancements.