Commercialization of Large-Scale Distributed Heterogeneous Fast Fourier Transform

There are many commercial and open source high-performance FFT implementations. However, they are typically restricted to single node systems and possibly multiple GPU systems. If the size of the problem does not fit within the CPU or GPU memory of a single node, users are left with very specialized or ad-hoc solutions. To serve this market segment, we will build a robust distributed (multi-node) heterogeneous (multi-GPU), FFT library. Communication bandwidth is the major bottleneck in FFT workloads (both between GPUs and nodes). In this proposal we use the CPUs as agents for asynchronously packing and unpacking the data between nodes and GPUs. They play the role of a complex memory reshaping and transposing engine. Further, we explore communication-avoiding methods methods such as oversampling and fast multipole methods to minimize the data transferred. In Phase I, we will evaluate bottlenecks of state-of-the-art code of high performance computing platforms such as supercomputers and Amazon Web Services. We will specify the FFT problem in a framework called Mathematics of Arrays (MoA), which will allow us to explore mathematically provable valid transformations of computation reordering and data reshapes and transposes. We will explore transformations that expose asynchrony such that we can hide the latency of all of the computation and provide enough runtime slack to absorb any isolated communication slowdowns. We will build a prototype FFT function using a distributed heterogeneous runtime system employing the derived MoA transformations and validate it against state-of-the-art high performance methods. Fourier transforms are used in a large number of industry and government applications. Efficient fast Fourier transform (FFT) codes are necessary in DOE applications such as particle applications (VPFFT), molecular dynamics (LAMMPS, NAMD), cosmological simulations (HACC), and applications in machine learning. In industry, FFT is employed in a large number of simulations including full wave inversion for geological studies and turbulence simulations for vehicle design. There is significant DOE and industry benefit to a FFT implementation that can scale to arbitrary sized memory footprints on both multi-node and multi-GPU systems.