An in situ, streaming, data- and compute-intensive platform for experimental data

Advances in sensors, experimental instrumentation, and computational technologies provide new opportunities for applied research and development in academia and industry. These same advances dramatic increase the volume, velocity, and variety, the three V’s of Big Data, in experimental data; and concurrently increases in the scale of computational resources required for processing this data. Applications that are both data- and compute-intensive are important tools supporting the discovery and innovation process. However, until recently, the development of the corresponding data-intensive and compute-intensive technologies have evolved along two independent paths. Born and matured before the Big Data era, high-performance computing (HPC), primarily scientific-oriented, applications have not been designed to leverage Big Data streaming frameworks. Recently emergence of Spark (http://spark.apache.org) platform has resolved many of the barriers to enable compute-intensive applications from within a data-intensive environment. While Spark is useful and gaining popularity in conventional data-analytics applications, its usage is limited in domain specific applications because of incompatible data formats, lack of fine-level interprocess communication, and support to use existing HPC code. Our proposed platform will provide an open-source, collaborative system that will extend Spark with workflows that enable academia and industry to utilize HPC applications while reducing or eliminating the need to perform data duplication and extract, transfer, load (abbreviated as ETL) as a prerequisite to utilizing an HTC environment. Also, we will minimize data transfer cost by applying in situ techniques and smart decisions on region and storage selection. Spark with MPI provides a complete HPC-HTC (High Throughput Computing) platform that is capable of data-intensive, compute-intensive processing needed for next generation analytics in domains such as Climate, High Energy Physics, Deep Learning, Experimental Data, and Computational Chemistry.