Adaptive MPI: Providing Dynamic RTS Support for Large-scale MPI Applications

High Performance Computing technology has advanced significantly. Advanced parallel computers are being built, with DOE leadership. Yet, the American engineering/manufacturing industry has not leveraged parallel computing at a significant level, partly because of the challenging nature of modern simulation software. US competitiveness in manufacturing may suffer due to its ineffective utilization of parallel computers. Objective: The broad objective of this project is to make Adaptive MPI (AMPI) become a plug-and-play replacement for other MPI implementations, but with added adaptive automatic capabilities such as load balancing, fault tolerance, and checkpoint/restart. This easy to state objective leads to a complex set of research/implementation challenges, but will ensure widespread adoption. Phase I Work: Included increased MPI standard compliance, performance improvements to match or exceed industry-standard implementations, exploration, enumeration and amelioration of ease-of-use concerns, and industry discussions and collaborations to identify challenges and opportunities. Phase II Plans: Includes a) improving value proposition by enhancing adaptivity features, flexible runtime-optimized integration of within-node parallelism, adaptively optimized support for cloud environments, etc., b) lowering barrier to entry, e.g by automating global-variable privatization for thread safety, standard compliance, Python support, etc., c) providing infrastructure enhancements, and d) application demonstrations with external partners, and in-house development of libraries and mini-applications. Commercial Applications and Other Benefits: The SBIR project effort is aimed at producing a commercially viable product based on Adaptive MPI that incorporates innovative research on dynamic run-time systems with the familiar programming interface of MPI. It will be the main commercial application to result directly from this effort. In addition, it is expected that libraries developed using the Adaptive MPI product will be an additional category of commercial applications. Some of these may come from Charmworks, Inc., but it is expected that many, in the future, will come from other companies. End-user applications is probably the largest category of commercial applications enabled by this project effort. This project aims to achieve a broad impact on the state of HPC adoption among the manufacturing and engineering industries.