QCFuse: Query-Centric Cache Fusion for Efficient RAG Inference
Summary: arXiv:2604.08585v1 Announce Type: cross
Abstract: Cache fusion accelerates the generation process of large language models (LLMs) equipped with Retrieval-Augmented Generation (RAG) through key-value (KV) caching and selective token recomputation. This innovation aims to reduce computational costs while improving efficiency. However, existing methods primarily rely on local perspectives for token selection, lacking the global awareness that user queries can provide. Utilizing this global awareness presents challenges due to the high costs associated with obtaining context-aware query representations and the strict pipeline constraints necessary for efficient attention analysis.
This article introduces QCFuse, an innovative KV cache fusion system centered on the user query. QCFuse leverages semantic summary anchors to enhance query representations and selectively recomputes query-related tokens to improve accuracy. This system updates tokens based on the attention distribution from the most critical Transformer layer, thereby preserving the high efficiency of the pipeline structure.
Key Features of QCFuse
- Global Awareness: QCFuse incorporates global context from user queries to optimize token selection, moving beyond local perspectives.
- Semantic Summary Anchors: By utilizing semantic anchors, QCFuse enhances the quality of query representations, leading to improved performance.
- Selective Token Recomputing: The system focuses on recomputing only the most relevant tokens, significantly reducing computational overhead.
- Attention Distribution Optimization: By updating tokens based on the attention distribution of the most critical Transformer layer, QCFuse maintains pipeline efficiency.
Performance Evaluation
Evaluations conducted on real-world datasets demonstrate that QCFuse significantly improves the response efficiency of LLMs by 40%, while maintaining equivalent accuracy compared to current methods. This is a noteworthy achievement in the domain of AI, where efficiency and accuracy are paramount. In certain scenarios, QCFuse also achieves an attention denoising effect, resulting in higher response accuracy.
Impact on LLM Inference Optimization
The introduction of QCFuse presents substantial potential in optimizing LLM inference processes. As AI continues to evolve, the demand for efficient and accurate models becomes increasingly critical. The innovative approach taken by QCFuse not only addresses the challenges posed by traditional methods but also sets a new standard for future developments in the field.
In conclusion, QCFuse stands as a promising advancement in the realm of large language models and retrieval-augmented generation. By centering its methodology around user queries and enhancing token selection through global awareness, it paves the way for more efficient and accurate AI systems.