Virtual HPC Cluster Provisioning in the Cloud

Because the Department of Energy (DOE) must address questions of national importance that do not admit to direct experimentation, high-performance computing (HPC) clusters provide critical simulation capability for the DOE mission. Indeed, HPC clusters are the primary computing resources used by most of the computational science community today. However, cluster expenditures generally provide a poor return on investment (ROI) when actual utilization and total cost of ownership are taken into account, yet a lack of alternatives has stimulated a proliferation of underutilized "closet clusters." Pay-as-you-go 'cloud computing' solutions are delivering unprecedented computational price-performance and accessibility improvements to web application customers and offer hope to cluster users as well, but to date cloud infrastructures have failed to provide the capabilities of an HPC environment in an on-demand fashion that would obviate the need to purchase physical clusters. Designed to integrate with Parabon's mature utility computing solution, which already boasts many of the capabilities sought under this topic, this project proposes to develop three fundamental software capabilities that are needed to enable virtual HPC clusters to be provisioned and used in a cloud: (1) the dynamic discovery of a cloud's inter-nodal network topology and point-to-point bandwidth capabilities to facilitate creation of virtual clusters with fast, secure interconnects; (2) automatic provisioning of and scheduling of jobs to an instantiated virtual cluster; and (3) dynamic re-provisioning of virtual machines on a cloud running at full capacity to service additional demand with only modest degradation of overall performance. Commercial Applications and Other Bene