The International Conference for High Performance Computing, Networking, Storage, and Analysis
The International Conference for High Performance Computing, Networking, Storage, and Analysis
Workshop: The 18th Workshop on Workflows in Support of Large-Scale Science (WORKS23) - Part 2 of 2
Authors: Dante D. Sanchez-Gallegos (University Carlos III of Madrid, Spain); J. L. Gonzalez-Compean (Cinvestav Tamaulipas); Jesus Carretero (University Carlos III of Madrid, Spain); and Heidy Marin-Castro (Cátedras CONACYT - Universidad Autónoma de Tamaulipas)
Abstract: This paper presents an adaptive continuum synchronisation method for data science pipelines deployed on edge-fog-cloud infrastructures. In a diagnostic phase, a model, based on the Bernoulli principle, is used as an analogy to create a global representation of bottlenecks in a pipeline. In a supervision phase, a watchman/sentinel cooperative system monitors and captures the throughput of the pipeline stages to create a bottleneck-stage scheme. In a rectification phase, this system produces replicas of stages identified as bottlenecks to mitigate the workload congestion using implicit parallelism and load balancing algorithms. This method is automatically and transparently invoked to produce in runtime a steady continuum dataflow. To test our proposal, we conducted a case study about the processing of medical and satellite data on fog-cloud infrastructures. The evaluation revealed that this method creates, without characterising workloads nor knowing infrastructure details, continuum dataflows, which yield a competitive performance with solutions in the state-of-the-art.