Accelerated In-Storage Data Mining of Light Sources

Modern X-ray and synchrotron light sources at Department of Energy (DOE) facilities generate vast datasets, often reaching petabyte scale. The growth in data volume and complexity presents significant challenges in processing, analyzing, storing, and managing these datasets efficiently. Addressing these challenges requires advanced tools that extract insights rapidly and securely while minimizing networking and storage bottlenecks. This project addresses these challenges by enhancing an analytical data platform to enable the rapid, in-place analytics on large image-rich datasets which are distributed across a large storage cluster on which the platform runs. In Phase I, we demonstrated the ability to partition large Hierarchical Data Format Version 5 (HDF5) datasets and to process them in parallel within this platform using User Defined Functions (UDFs), including AI inference models. We also validated the platform’s compatibility with tools commonly used by DOE researchers for accessing remote storage and computational resources. Phase II will expand the platform's capabilities to analyze a broader range of image-rich scientific and commercial datasets, improve its stability, security, and performance, and prepare the solution for commercial trials. The enhanced platform is expected to reduce analysis time for large image-rich datasets by up to 100x by leveraging massively parallel processing. This capability will empower researchers to derive actionable insights more quickly and efficiently, accelerating scientific discovery. Potential commercial applications include the analysis of drone surveillance data, with future opportunities in advanced manufacturing, medical imaging, and environmental monitoring. By addressing the growing demand for scalable analytics, the platform will foster innovation, enhance research efficiency, and support the development of data-intensive technologies.