Research Posters Archive

Balancing Latency and Throughput of Distributed Inference by Interleaved Parallelism

Authors: Jiangsu Du, Jinhui Wei, and Jiazhi Jiang (Sun Yat-Sen University); Shenggan Cheng (National University of Singapore); and Zhiguang Chen, Dan Huang, and Yutong Lu (Sun Yat-Sen University)

Abstract: Distributed large model inference is still in a dilemma where balancing latency and throughput, or rather cost and effect. Tensor parallelism, while capable of optimizing latency, entails a substantial expenditure. Conversely, pipeline parallelism excels in throughput but falls short in minimizing execution time.

To address this challenge, we introduce a novel solution - interleaved parallelism. This approach interleaves computation and communication across requests. Our proposed runtime system harnesses GPU scheduling techniques to facilitate the overlapping of communication and computation kernels, thereby enabling this pioneering parallelism for distributed large model inference. Extensive evaluations show that our proposal outperforms existing parallelism approaches across models and devices, presenting the best latency and throughput in most cases.

Best Poster Finalist (BP): no

Poster: PDF
Poster summary: PDF


Back to Poster Archive Listing