Tag: LLM inference
Browse our exclusive articles!
Cost-Efficient LLM Inference with Token-Budget Routing
Optimize LLM inference costs using token-budget-aware pool routing to reduce GPU usage and save millions annually in AI deployments.
Performance and Energy Trade-offs in Multi-Request LLM Workflows
Explore how multi-request workflows impact large language models' performance and energy use, with strategies to optimize latency and efficiency.
Energy-Efficient LLM Inference on GPUs: Watt Counts Benchmark
Discover Watt Counts, the largest energy-aware benchmark for sustainable LLM inference across heterogeneous GPU architectures, reducing energy use up to 70...
CSAttention: Fast, Accurate Sparse Attention for LLMs
Discover CSAttention, a novel sparse attention method boosting LLM inference speed by 4.6x while maintaining accuracy with high sparsity and long contexts.
KV Cache Management Strategies for Efficient LLM Inference
Explore and compare KV cache management strategies to optimize memory use and boost performance in large language model inference tasks.