Terabits Data Transfer Toolset for High-End Distributed BES Data Analytics

The NSLS-II, APS, ALS, and LCLS are planning for or in the process of upgrading their light source facilities to increase beam brightness and incorporate advanced detector tech- nologies. These upgrades will significantly enhance image resolution and expand data volume, necessitating new high-speed tools for transferring and streaming scanning to local/remote data centers for monitoring, data processing, and real-time progress tracking. We propose to develop data transfer and streaming tools that achieve one Terabit-per-second (1Tbps), significantly exceeds the capabilities of existing data transfer tools, and meet the following key requirements: Compact, High-Efficiency Appliance: A streamlined 2U rack-mounted appliance, CPU Offloading for Enhanced Efficiency: The system will offload data prepro- cessing tasks, such as compression, encryption, and decryption, from the CPU, freeing it to handle other critical operations, Seamless Integration with High-Performance Distributed File Systems: ensuring easy adoption by existing high-speed storage infrastructures, and Versatile Data Transfer Capabilities: Support offline file transfers and real-time data streaming across WAN. To achieve these objectives, we leverage OS libraries, paral- lel processing techniques, high-speed network protocol design, and cutting-edge hardware accelerators. We propose an asynchronous, memory-centric software framework designed for high-throughput, data-intensive applications. This framework will incorporate a zero-copy data movement system using asynchronous Re- mote Direct Memory Access (RDMA), maximizing data preprocessing and transmission parallelism. Our software framework is optimized for integration with Data Processing Units (DPUs) and utilizes an asyn- chronous API and specialized libraries to assign data processing tasks to DPU accelerators. Our RDMA-1000 achieved file transfer at the speed of 399 Gbps on NVIDIA BlueField-3 DPUs, utilizing over 99% of their 400 Gbps network capacity over LAN. Our software demon- strated 75% bandwidth utilization of the 100 Gbps network on the DOE ESnet testbed. The streaming version of RDMA-1000 successfully transmitted NSLS-II X-ray video frames at 40 Gbps. In Phase II, we will continue improving our RDMA-1000 system for ter- abit (1Tbps) data transfer. In addition, we will collaborate closely with the beamline scientists at Brookhaven National Laboratory (BNL) to co-design a high-performance data transfer solution capable of efficiently moving multi-dimensional scanning images from the beamline across ESnet to the NERSC for near real- time experiment monitoring and control. This co-design process will ensure that the solution aligns with the operational needs of NSLS-II, enabling rapid adoption across DOE’s major photon science facili- ties, including ALP, APS, and LCLS. This collaboration also aims to meet the diverse data needs of DOE high-energy and nuclear physics and commercial clouds.