Efficient High-Performance Data Transfer Over ESnet for Massive-Scale Data Analytics

DOE BES facilities are expected to increase data rates by several orders of magnitude this decade. For example, raw data throughput at the Linac Coherent Light Source (LCLS) is expected to generate analyzable data at 70 TeraBytes (TB) per hour. Such data volumes require advanced means to transfer to High-Performance Computing (HPC) centers for processing or long-term storage. We are proposing the development of a novel Data Transfer and Streaming Node (DTSN) server to enable the exascale data movement demands of DOE Basic Energy Science (BES) facilities. The DTSN will be compatible with current DOE data transfer technology developed by the DOE Energy Sciences Network (ESNet), which utilizes so called Data Transfer Node (DTN) computers but will increase speed and decrease power consumption using Data Processing Unit (DPU) hardware and the User Datagram network Protocol (UDP). We will develop a data transmitter computer which will serve as the testbed to develop DTSN transmission software. We will develop a prototype DTSN which receives streams from the data transmitter. We will save >160 Gbps data streams directly to a Fast Feedback Storage system using direct storage access. Future applications will include X-ray science, Basic Energy Science, nuclear physics, high energy physics, nuclear astrophysics, homeland security, and education. Public will benefit from fast data transfer helping fundamental research conducted at the National Laboratories and research institutions.