Tag: speculative decoding
Browse our exclusive articles!
ECHO: Fast Speculative Decoding for High-Concurrency LLMs
ECHO boosts large language model inference with elastic speculative decoding and sparse gating, achieving up to 5.35x speedup in high-concurrency scenarios...
SPEED-Bench: Benchmarking Speculative Decoding for LLMs
Discover SPEED-Bench, a unified benchmark for evaluating speculative decoding in large language models with diverse, real-world workloads and production in...
SpecMoE: Fast, Efficient Mixture-of-Experts Inference
Discover SpecMoE, a memory-efficient MoE inference system boosting throughput by 4.3x with self-assisted speculative decoding for scalable AI models.
LongSpec: Efficient Lossless Speculative Decoding for Long Contexts
Discover LongSpec, a novel framework for lossless speculative decoding that boosts long-context AI model speed and efficiency with reduced memory use.