Tag: KV Cache Compression
Browse our exclusive articles!
AdapShot: Efficient Adaptive Many-Shot In-Context Learning
Discover AdapShot, a novel framework boosting Many-Shot In-Context Learning efficiency with adaptive shot optimization and semantic KV cache reuse.
Predictive Multi-Tier KV Cache Memory for GPU Inference
Optimize large-scale GPU inference with predictive multi-tier KV cache memory management, boosting performance and cutting costs significantly.
OjaKV: Efficient Online KV Cache Compression for LLMs
Discover OjaKV, a context-aware online low-rank KV cache compression method that boosts large language model efficiency and memory usage.
Sequential KV Cache Compression Beyond Shannon Limit
Discover advanced sequential KV cache compression using probabilistic tries, surpassing the per-vector Shannon limit for transformer models.
LoopGuard: Stop Repetition Loops in AI Text Generation
LoopGuard breaks self-reinforcing attention loops in AI models, reducing repetition and enhancing text diversity with dynamic KV cache intervention.