An integrated platform for high performance data management and analysis at X-ray light sources

X-ray light sources enable commercial research and development and scientific discovery in many disciplines, from medicine to engineering to historical preservation – producing thousands of publishable discoveries every year. As a result of their success, facilities serve an increasingly diverse user community, often providing concurrent beam time and user support to dozens of domain scientists with unique backgrounds. Without significant software and infrastructure support, end users are faced with the challenge of adapting their workflows to coordinate experimental execution, data analyses, and simulations to facility- specific resources. An open-source platform for integrated data management, analysis, and simulation will be developed in support of end users and facility scientists at user facilities. The platform will enable selection and manipulation both experimental and simulation data by end users. The platform will support the staging of data on local and remote computing nodes, leveraging community frameworks for scientific data to expedite in memory data operations. A corresponding graphical user interface will permit the templating of additional operations and analyses. These tools will improve simulation and analysis workflows while reducing barriers to entry for high performance computing resources. The proposed work will develop a browser-based graphical user interface for data management. The interface will leverage an existing framework for orchestrating experimental controls and data acquisition to provide enhanced data transport and analysis capabilities for users. The interface will permit the selection, staging and transport of data between local and remote machines via a generic data model. The interface will enable users to access experimental and simulation data, and transfer them to remote systems for further analysis. The interface will be demonstrated with both simulation and measurement data transport between remote clusters for analysis, before being deployed and tested at an operating X-ray beamline. X-ray light sources serve an increasingly diverse, worldwide user community of scientists and engineers across many disciplines, with applications in basic research and industry. The open-source scientific application framework developed during this work will provide user support for sophisticated experimental configurations, data acquisition, analysis, and simulation workflows. The inclusion of a performance oriented data framework will enable operation across distributed computing resources. This technology will be leveraged to deliver value to light source facilities and their growing community of users.