Birds of a Feather

Workflows Community: Modern Workflows for Continuum and Cross-Facility Computing

Authors: Rafael Ferreira da Silva (Oak Ridge National Laboratory (ORNL)), Kyle Chard (University of Chicago), Jorge Ejarque (Barcelona Supercomputing Center (BSC)), Olivier Terzo (LINKS Foundation Inc), Debbie Bard (Lawrence Berkeley National Laboratory (LBNL)), Frédéric Suter (Oak Ridge National Laboratory), Tom Gibbs (NVIDIA Corporation), Hal Finkel (DOE Office of Advanced Scientific Computing Research), Alberto Scionti (LINKS Foundation Inc), Iacopo Colonnelli (Università di Torino)

Abstract: Discoveries in science increasingly rely on workflows to coordinate complex experiments, ranging from cloud-based data preprocessing to multi-facility computational workflows. Continuum and cross-facility workflows have gained prominence, providing continuous computing access and spanning multiple sites. This BoF session, organized by the Workflows Community Initiative, will address challenges, opportunities, and future directions for continuum and cross-facility workflows. Participants will share domain-specific insights, covering topics such as facility coordination, metadata tracking, and standardization. The BoF will produce tangible outputs, including lightning talks and a community roadmap, fostering networking and international collaborations.

Long Description: Science discovery relies increasingly on workflows to coordinate large and complex scientific experiments, ranging from cloud-based data preprocessing pipelines to multi-facility instrument-to-edge-to-HPC computational workflows. Continuum and cross-facility workflows have become prominent in computational sciences. Continuum workflows represent analysis pipelines that require continuous computing access, while cross-facility workflows span multiple sites, encompassing experiments and computing facilities. Cross-facility workflows offer resiliency for real-time workflows and can be viewed as a solution for the needs of continuum workflows. It is also important to consider continuum and cross-facility computing from a data perspective, ensuring workflow systems can handle different representations and storage systems.

The inaugural BoF at SC22 focused on integrating HPC and AI workflows and featured talks from representatives of the U.S. DOE and NSF research offices and leading researchers in the field. The outcomes of this BoF shaped the discussion topics for the annual virtual Workflows Community Summit (https://workflows.community/summits/2022/).

In the second edition of this BoF, the Workflows Community Initiative (https://workflows.community) will convene to discuss challenges, opportunities, research directions, and future pathways for continuum and cross-facility workflows. The perspectives of scientists as users of workflow systems will also be included, with users providing insights into the challenges specific to their science domains. Topics for discussion will encompass coordination and cooperation among computing and experimental facilities, metadata tracking across different sites, co-design of experiment facilities, standardization of task descriptions, edge-to-cloud continuum and data exchange, adaptive compression techniques, and the definition of common terms, building blocks, and concepts in workflows.

This BoF will yield tangible outputs, including lightning talks, live-poll questions and responses, and a draft of an updated community roadmap. Participants will be encouraged to join the Workflows Community Initiative to facilitate networking and foster potential international collaborations beyond the conference.

The Supercomputing conference serves as an ideal platform for this BoF, bringing together major HPC resource providers and scientific computing users. It offers an opportunity to discuss existing workflow capabilities supported by international HPC centers, stimulate conversations on implementation and adoption, and guide future collaborative efforts.

Website: https://workflows.community/bof/sc23