HybridKV: Hybrid KV Cache Compression for Efficient Multimodal Large Language Model Inference
Summary: arXiv:2604.05887v1 Announce Type: new
Abstract
Multimodal Large Language Models (MLLMs) have advanced unified reasoning over text, images, and videos, but their inference is hindered by the rapid growth of key-value (KV) caches. Each visual input expands into thousands of tokens, causing caches to scale linearly with context length and remain resident in GPU memory throughout decoding. This results in prohibitive memory overhead and latency, even on high-end GPUs.
A common solution is to compress caches under a fixed allocated budget at different granularities: token-level uniformly discards less important tokens, layer-level varies retention across layers, and head-level redistributes budgets across heads. However, these approaches stop at allocation and overlook the heterogeneous behaviors of attention heads that require distinct compression strategies.
Introducing HybridKV
We propose HybridKV, a hybrid KV cache compression framework that integrates complementary strategies in three stages:
- Classification of Heads: Heads are first classified into static or dynamic types using text-centric attention.
- Top-Down Budget Allocation: A hierarchical scheme assigns KV budgets.
- Compression Techniques: Static heads are compressed by text-prior pruning and dynamic heads by chunk-wise retrieval.
Performance Evaluation
Experiments conducted on 11 multimodal benchmarks with the Qwen2.5-VL-7B model demonstrate that HybridKV significantly enhances performance. The key findings include:
- Reduction of KV cache memory by up to 7.9 times.
- Achieving 1.52 times faster decoding.
- Maintaining performance levels nearly equivalent to or even surpassing that of the full-cache MLLM.
Conclusion
HybridKV presents a promising solution to the challenges faced by MLLMs in terms of memory overhead and latency during inference. By leveraging a hybrid approach to KV cache compression, it addresses the unique characteristics of attention heads, thereby improving overall efficiency without compromising performance. This advancement could pave the way for more efficient multimodal applications in various domains, including natural language processing, computer vision, and beyond.