Terabits Data Transfer Toolset for High-End Distributed BES Data Analytics

Statement of the Problem. Our main objective is to facilitate the processing of large amounts of data for ongoing experiments in clouds and DOE supercomputers. We also strive to address the following challenges to ensure that data analysis is performed synchronously while experimental data are being generated and streamed. APS, ALS, and LCLS are upgrading facilities to improve beam brightness and significantly increase image resolution and data volume (10PB data per year). The TCP/IP-based data transfer tools commonly used in storage clouds and the Internet fall far behind the available network bandwidth and often constitute an end-to-end bottleneck. Current data transfer protocols such as Globus, SCP/sFTP, GridFTP, and BigData Express used at DOEĺs NERSC, ORLCF, ALCF do not address this critical bottleneck in their end systems and usually use a large number of data transfer nodes (DTNs) to scale up the aggregated transfer performance. This approach requires a large footprint for DTN, consumes too much electricity, and results in unsatisfactory performance for DOE basic energy science applications with new transfer requirements. How the problem is Being Addressed. Sunrise Technology Inc. proposes to develop a Terabits data transfers/streaming protocol optimized for end-systems with throughput performance that far outstrips what is capable of TCP/IP. We will implement a high-speed communication middleware layer based on Remote Direct Memory Access (RDMA), which is the common substrate to accelerate various data transfer tools, such as FTP, HTTP, file copy, sync, and remote file I/O. This middleware offers better end-to-end bandwidth performance than traditional TCP-based alternatives, while it hides the heterogeneity of the underlying high-speed architecture. We provide a reference implementation of the popular file-transfer protocol over the RDMA-based middleware layer and commercial-of-the-shelf network cards that support RDMA over Converged Ethernet Version 2 (RoCE v2). The advantage of our protocol is that the generated data packets are fully compatible with the Internet, eliminating the need for network provisioning, and can be downloaded as a user app on physicistsĺ desktops or computer clusters. To enforce DOE Cybersecurity requirements, we use Globus Grid certificate to authenticate and authorize users to access datasets. What is to be done in Phase I. We envision four tasks in this proposal: 1) Design the core of our middle- ware software that offers data transfer, real-time data streaming and online data analysis, and access primitives, supports multithreaded architecture, and facilitates multistream data transfer to exploit the parallelism of RDMA operations. 2) Implement buffer management and task synchronization mechanisms. This will allow for maximum buffer reuse, reduce synchronization expense, and minimize RDMA connection configuration costs. 3) Combine NVIDIA Jetson Orin (ARM-based CPU) with NVIDIA ConnectX-5 (2x100 Gbps) to create an energy-efficient solution and scale it to terabit transfer with a hardware footprint of only 2 4 rack units. 4) Integrate our data transfer tools with TILED. TILED is a data search and access service for BES datasets. We integrate our data transfer tools with TILED to provide data search and access service for BES datasets and support data discovery, retrieving, and replications among NSLS-II facilities and DOE ASCR Leadership Computer Facilities (LCF). Sunrise Technology will collaborate with Dr. Hanfei Yan in Brookhaven National Laboratory and address topic C57-09 Subtopic b) Efficient High-Performance Data Transfer over ESnet for Massive-Scale Data Analytics. Commercial Applications and Other Benefits Supercomputers that generate large simulation data, government agencies with surveillance data, finance (FinTech) institutes with big data analysis for investment decisions, and cloud centers that frequently transfer large amounts of data for time-sensitive applications and migrate computing jobs and intermediate results.