The International Conference for High Performance Computing, Networking, Storage, and Analysis
The International Conference for High Performance Computing, Networking, Storage, and Analysis
Workshop: XLOOP 2023: The 5th Annual Workshop on Extreme-Scale Experiment-in-the-Loop Computing
Authors: Hannah Parraga, John Hammonds, Steven Henke, Siniša Veseli, William Allcock, Benoit Côté, Ryan Chard, Suresh Narayanan, and Nicholas Schwarz (Argonne National Laboratory)
Abstract: This paper explores the challenges and solutions for managing the vast amount of data generated by the Advanced Photon Source (APS), a synchrotron light source producing ultra-bright x-rays for diverse scientific domains. With 68 experimental beamlines, the APS serves a wide user base across academia, government, and industry. The ongoing upgrade of the APS storage ring and new instruments will amplify data generation and processing demands. This paper discusses the approach to address these demands through automated standardized workflows for faster scientific insights. The APS Data Management System coordinates data related tasks and interfaces with Globus. Through integration with the Argonne Leadership Computing Facility (ALCF), APS users can efficiently access HPC resources. Standardized workflows have led to reduced computational burdens on scientists and greater accessibility of HPC resources. We demonstrate how standardization and collaboration enable scientists to rapidly produce meaningful scientific results, establishing a streamlined path from collection to publication.