KV Packet: Recomputation-Free Context-Independent KV Caching for LLMs
Summary: arXiv:2604.13226v1 Announce Type: cross
Abstract
Large Language Models (LLMs) have become an integral part of various applications, but their performance is often constrained by inference latency. One of the primary methods to address this issue is through Key-Value (KV) caching. Traditional KV caches, however, present a significant challenge: they are context-dependent. This means that when a cached document is reused in a different context, it necessitates the recomputation of KV states to adjust for changes in attention distribution. In the quest for efficiency, several existing solutions, including CacheBlend, EPIC, and SAM-KV, have been developed to mitigate this problem. These methods aim to selectively recompute only a subset of tokens, but they still incur considerable computational overhead, leading to increased Time-to-First-Token (TTFT) latency.
Introduction to KV Packet
In response to the limitations of current caching mechanisms, we introduce KV Packet, a novel recomputation-free cache reuse framework. This innovative approach conceptualizes cached documents as immutable “packets” that are enveloped in lightweight trainable soft-token adapters. These adapters are designed to be trained via self-supervised distillation, effectively bridging the gaps created by context discontinuities.
Key Features of KV Packet
- Recomputation-Free: KV Packet eliminates the need for recomputation, drastically reducing the computational demands associated with cache reuse.
- Immutable Packets: Cached documents are treated as fixed entities, ensuring consistent performance across varying contexts.
- Lightweight Soft-Token Adapters: The use of trained soft-token adapters allows for efficient adaptation to new contexts without the overhead of full recomputation.
- Self-Supervised Distillation: This technique enables the system to learn and improve over time, enhancing the bridging of attention distributions across contexts.
Performance Evaluation
To assess the effectiveness of the KV Packet framework, we conducted extensive experiments using the Llama-3.1 and Qwen2.5 models. The results indicate that KV Packet achieves near-zero Floating Point Operations per Second (FLOPs), significantly lowering TTFT compared to traditional recomputation-based baselines. Notably, while maintaining these advantages, KV Packet also preserves F1 scores that are comparable to those obtained from full recomputation methods.
Conclusion
In conclusion, KV Packet represents a significant advancement in the field of LLMs by addressing the challenges posed by context-dependent KV caching. With its innovative approach and demonstrated performance benefits, KV Packet paves the way for more efficient and responsive language models, ultimately enhancing user experience and application performance in real-world scenarios.