StreamServe: Adaptive Speculative Flows for Low-Latency Disaggregated LLM Serving
Summary: arXiv:2604.09562v1 Announce Type: cross
Abstract: Efficient LLM serving must balance throughput and latency across diverse, bursty workloads. We introduce StreamServe, a disaggregated prefill decode serving architecture that combines metric aware routing across compute lanes with adaptive speculative decoding that tunes speculation depth online from runtime signals.
StreamServe comprises four components: StreamScheduler for request orchestration, FlowGuard for multi signal routing, PipeServe Engine for disaggregated prefill decode execution on multi GPU, and SpecuStream for runtime adaptive speculation. We evaluate StreamServe on four benchmarks ALPACA, GSM8K, HUMANEVAL, and SUM with 80 queries each and 320 total using 4 A800 40GB GPUs configured as two stream pairs.
Across these workloads, StreamServe reduces latency by 11 to 18 times relative to tensor parallel vLLM baselines and reaches throughput up to 2235 tokens per second on summarization tasks. Time per output token remains stable across configurations, indicating that the gains arise from architectural efficiency rather than token quality degradation. Although evaluated on a single node 4 GPU setup, these results suggest that jointly adapting routing and speculation within a disaggregated framework creates a distinct operating regime for LLM inference.
Key Components of StreamServe
- StreamScheduler: This component is responsible for orchestrating requests, ensuring that workloads are managed efficiently across available resources.
- FlowGuard: A multi-signal routing mechanism that optimizes the flow of data based on real-time metrics, enhancing the overall performance of the system.
- PipeServe Engine: Designed for disaggregated prefill decode execution, this engine utilizes multiple GPUs to process data in a streamlined manner.
- SpecuStream: This innovative feature allows for runtime adaptive speculation, adjusting the speculation depth based on ongoing runtime signals to optimize performance.
Performance Evaluation
StreamServe was evaluated using four distinct benchmarks: ALPACA, GSM8K, HUMANEVAL, and SUM. Each benchmark consisted of 80 queries, culminating in a total of 320 queries processed by the system. The evaluation was conducted on a setup featuring four NVIDIA A800 40GB GPUs, configured into two stream pairs.
The results from the evaluation were promising. StreamServe significantly outperformed tensor parallel vLLM baselines, achieving a reduction in latency ranging from 11 to 18 times. In terms of throughput, the system was able to handle up to 2235 tokens per second, particularly excelling in summarization tasks. Notably, the time taken per output token remained consistent across various configurations, suggesting that the improvements in performance were attributable to the architectural efficiency of StreamServe rather than any compromises in the quality of the generated tokens.
Conclusion
While the evaluation was performed on a single node configuration with four GPUs, the findings indicate that StreamServe has the potential to revolutionize LLM inference. By integrating dynamic routing and speculative decoding within a disaggregated framework, StreamServe creates a unique operational environment that enhances both throughput and latency. This innovative approach may pave the way for future advancements in efficient LLM serving, particularly in handling diverse and bursty workloads.