Tag: LLM inference
Browse our exclusive articles!
Ragged Paged Attention: Fast LLM Inference Kernel for TPU
Discover Ragged Paged Attention, a high-performance LLM inference kernel optimized for TPU, boosting efficiency and reducing costs in large language model...
Vec-LUT: Fast Ultra-Low-Bit LLM Inference on Edge Devices
Discover Vec-LUT, a vector table lookup method boosting ultra-low-bit LLM inference speed up to 4.2x on edge devices with optimized memory usage.
SpecBranch: Boosting LLM Speed with Hybrid Speculative Decoding
Discover SpecBranch, a novel hybrid speculative decoding method that improves large language model inference speed by up to 4.5× with rollback-aware branch...
Boost LLM Inference Speed with Speculative Decoding on AWS
Enhance large language model inference using speculative decoding on AWS Trainium with vLLM for faster, cost-effective AI performance.
RPRA: Efficient LLM-Judge Prediction for Better Inference
Discover how RPRA improves LLM efficiency by predicting judge scores, boosting smaller models' performance without heavy computation.