The International Conference for High Performance Computing, Networking, Storage, and Analysis
The International Conference for High Performance Computing, Networking, Storage, and Analysis
Workshop: XLOOP 2023: The 5th Annual Workshop on Extreme-Scale Experiment-in-the-Loop Computing
Authors: Michael Prince, Doğa Gürsoy, Dina Sheyfer, and Ryan Chard (Argonne National Laboratory (ANL)); Benoit Côté (Argonne National Laboratory); Hannah Parraga and Barbara Frosik (Argonne National Laboratory (ANL)); Jon Tischler (Argonne National Laboratory); and Nicholas Schwarz (Argonne National Laboratory (ANL))
Abstract: In February and April 2023 live, at-scale data processing demonstrations were conducted between the Advanced Photon Source (APS), a synchrotron light source, and the Argonne Leadership Computing Facility (ALCF). These tests were run as part of a novel beamline technique: coded aperture laue micro-diffraction. This technique requires a significant amount of compute to decode appeture patterns embedded in the detector stream. An autonomous system was able to send data to ALCF during an experiment, utilize 50 nodes of the Polaris supercomputer to process 6-12 hour scans, and return the data back to the APS within 12-15 minutes behind the detector. With scan points arriving every 72 seconds, the system kept up with the beamline, potentially enabling in-experiment analysis. The data processing system utilizes Globus infrastructure and an on-demand queue to dynamically acquire nodes on Polaris. The underlying reconstruction algorithms were parallelized via MPI and accelerated with custom CUDA kernels.