Accelerating PETSc through Next-Generation Heterogeneous Supercomputing

It is often found that seemingly distinct, scientic codes in elds such as biology, engineering, and physics are faced with the same computational problemthey require the solution of sparse-linear systems arising from the discretization of elliptic and/or parabolic partial dierential equations. This bottleneck can severely hinder the application scientists ability to solve the most challenging problems if not properly implemented. Moreover, this problem becomes all the more pernicious as computing facilities evolve from traditional, homogeneous supercomputers composed of CPU-only nodes, to heterogeneous systems where accelerators, such as GPUs, are coupled to CPU nodes. Currently though, many commonly-used libraries of importance to the Department of Energy (DoE), lack the tools to take advantage of these emerging computing architectures. We propose to develop implementations of core computations (kernels) of Krylov algorithms for solving sparse linear systems on the most widespread and easy-to-use, heterogeneous computing architectures: multi-GPU machines. Then, we will build interfaces between the developed kernels that reside on the GPU with the widely-used, DoE-funded PETSc (Portable, Extensible Toolkit for Scientic Computation) library so that this new capability can be used transparently by an application scientist. We will optimize the core mathematical kernels and provide a robust set of preconditioner techniques in order that the solvers achieve fast convergence. Fast-convergence in the solvers reduces the time-to-solution in the application code thus increasing their productivity. We built high-performance versions of key algorithms and integrated them into the development version of PETSc. We also identied performance bottlenecks in the parallel-GPU implementation and developed a plan for addressing these issues. We will optimize the parallel-GPU performance in PETSc and integrate a comprehensive set of GPU-enabled preconditioner algorithms and solvers. We will then use the new GPU capabilities in the M3D-C1 fusion simulation code from Princeton Plasma Physics Laboratory on current and next-generation GPU clusters such as the to-be-built Titan machine at Oak Ridge National Lab. Commercial Applications and Other Benets: Successful completion of this project will enhance the Tech-X GPULib commercial library. In addition, this project will greatly increase the utility and scope of the DoE-funded PETSc library by providing a new capability to the much-broader scientic computing community. Moreover, since the PETSc library is an integral component of other Tech-X simulation software products, we expect that this technology will expand marketability and thus contribute to increased sales. Finally, studies performed during this project will be made available to the scientic computing community via peer-reviewed journal articles and conference presentations. It is often found that seemingly distinct, scientic codes in elds such as biology, engineering, and physics are faced with the same computational problemthey require the solution of sparse-linear systems arising from the discretization of elliptic and/or parabolic partial dierential equations. This bottleneck can severely hinder the application scientists ability to solve the most challenging problems if not properly implemented. Moreover, this problem becomes all the more pernicious as computing facilities evolve from traditional, homogeneous supercomputers composed of CPU-only nodes, to heterogeneous systems where accelerators, such as GPUs, are coupled to CPU nodes. Currently though, many commonly-used libraries of importance to the Department of Energy (DoE), lack the tools to take advantage of these emerging computing architectures. We propose to develop implementations of core computations (kernels) of Krylov algorithms for solving sparse linear systems on the most widespread and easy-to-use, heterogeneous computing architectures: multi-GPU machines. Then, we will build interfaces between the developed kernels that reside on the GPU with the widely-used, DoE-funded PETSc (Portable, Extensible Toolkit for Scientic Computation) library so that this new capability can be used transparently by an application scientist. We will optimize the core mathematical kernels and provide a robust set of preconditioner techniques in order that the solvers achieve fast convergence. Fast-convergence in the solvers reduces the time-to-solution in the application code thus increasing their productivity. We built high-performance versions of key algorithms and integrated them into the development version of PETSc. We also identied performance bottlenecks in the parallel-GPU implementation and developed a plan for addressing these issues. We will optimize the parallel-GPU performance in PETSc and integrate a comprehensive set of GPU-enabled preconditioner algorithms and solvers. We will then use the new GPU capabilities in the M3D-C1 fusion simulation code from Princeton Plasma Physics Laboratory on current and next-generation GPU clusters such as the to-be-built Titan machine at Oak Ridge National Lab. Commercial Applications and Other Benets: Successful completion of this project will enhance the Tech-X GPULib commercial library. In addition, this project will greatly increase the utility and scope of the DoE-funded PETSc library by providing a new capability to the much-broader scientic computing community. Moreover, since the PETSc library is an integral component of other Tech-X simulation software products, we expect that this technology will expand marketability and thus contribute to increased sales. Finally, studies performed during this project will be made available to the scientic computing community via peer-reviewed journal articles and conference presentations.