Cactus: Accelerating Auto-Regressive Decoding with Constrained Acceptance Speculative Sampling
Summary: arXiv:2604.04987v1 Announce Type: cross
Abstract: Speculative sampling (SpS) has been successful in accelerating the decoding throughput of auto-regressive large language models by leveraging smaller draft models. SpS strictly enforces the generated distribution to match that of the verifier LLM. This is unnecessarily restrictive as slight variations of the verifier’s distribution, such as sampling with top-$k$ or temperature, would also be acceptable. Typical acceptance sampling (TAS) alleviates this issue by accepting more tokens using entropy-based heuristics. However, this approach distorts the verifier distribution, potentially degrading output quality when the verifier encodes critical information. In this work, we formalize the speculative sampling algorithm through the lens of constrained optimization. Based on this formulation, we propose Cactus (constrained acceptance speculative sampling), a method that guarantees controlled divergence from the verifier distribution and increasing acceptance rates. Empirical results across a wide range of benchmarks confirm the effectiveness of our approach.
Introduction
The growing demand for efficient decoding processes within auto-regressive language models has led researchers to explore various methodologies aimed at improving throughput. Among these methodologies, speculative sampling (SpS) has emerged as a promising technique, particularly when combined with smaller draft models that can accelerate the overall decoding process. However, the traditional implementation of SpS imposes strict constraints that may hinder its effectiveness.
Challenges with Current Methods
While speculative sampling has shown potential, its strict adherence to matching the verifier large language model (LLM) distribution has raised concerns. The limitations of SpS can be summarized as follows:
- Restrictive Sampling: The requirement for the generated distribution to exactly match that of the verifier LLM can limit the diversity of outputs.
- Output Quality: In scenarios where critical information is encoded within the verifier, any distortion of the distribution could lead to a degradation in output quality.
- Acceptance Rates: The typical acceptance sampling (TAS) method, while increasing acceptance rates, does so at the cost of altering the verifier distribution.
Introduction of Cactus
To address these challenges, the research team has developed a new approach named Cactus (constrained acceptance speculative sampling). Cactus is grounded in the principles of constrained optimization, allowing for a more flexible sampling process while maintaining control over the divergence from the verifier distribution. Key features of Cactus include:
- Controlled Divergence: Cactus ensures that any divergence from the verifier distribution is kept within acceptable limits, thus preserving the integrity of critical information.
- Increased Acceptance Rates: By optimizing the sampling process, Cactus achieves higher acceptance rates without compromising on the quality of generated outputs.
- Empirical Validation: Extensive testing across various benchmarks has demonstrated the effectiveness of Cactus in enhancing decoding throughput.
Conclusion
The introduction of Cactus marks a significant advancement in the field of auto-regressive language model decoding. By addressing the limitations of current methods, this new approach not only accelerates throughput but also ensures the quality of generated outputs remains high. As the demand for efficient and effective language models continues to grow, innovations like Cactus will play a crucial role in shaping the future of natural language processing.