E4S: Extreme-Scale Scientific Software Stack for Commercial Clouds

C53-02b-271168The software used in High Performance Computing (HPC) and ArtificialIntelligence/MachineLearning (AI/ML) workloads is increasingly complex to maintain, install, and optimize. More problematic is the poor performance portability of applications between platforms, forcing site-specific re-engineering of codes. Existing solutions to deployment of AI/ML work?ows on commercial cloud environments are platform-specific, preventing migration from one cloud provider to another. This project proposes to address the problem by combining the use of an integrated stack of HPC software, which provides multi-platform container images, with a highly performant and performance-portable Message Passing Interface (MPI) library for fast, inter-and intra- node communication on four cloud platforms, made available through a platform-neutral interface. Phase I has evaluated the feasibility of this solution and built prototypes for evaluation. We made improvements to an MPI implementation to support high-performance deployments of MPI applications on cloud platforms; built high-performance versions of commonly usedDeep Learning frameworks for cloud deployment; made use of high-speed network adapters and accelerators within the cloud environments; and evaluated the use of a platform-neutral web interface for deployment on a cloud vendor. In Phase II, the proof-of-concept developed in Phase I will be ported to three additional cloud platforms, and the high-performance MPI implementation will be tuned for each of these platforms. All four supported plat- forms will be made available through a single platform-neutral interface. The ability to use the Phase II product to easily port an application from traditional HPC clusters to all four cloud platforms will be demonstrated. The success of the project will deliver a productive platform for transitioning important HPC applications (many developed in National Laboratories) to more accessible cloud based HPC platforms in a portable manner while retaining high performance. It will be beneficial to practically all scalable HPC applications ranging from modeling and simulation to AI/ML, where advance message communication hardware and access to accelerator technologies are being more commonly supported in commercial cloud systems. In particular, data analytics and deep learning are areas of high growth and of benefit to a broad range of industries. High performance is critical for these codes — a poorly performing code wastes compute resources, preventing purchased hardware from being used for other uses, increasing a business’s costs for cloud computing resources, and increasing time to solution.