LLM Inference Degradation in Production: KV-cache, OOM, and the p99 Tail

VK Cloud technology evangelist Stas Pogorzhelsky explains that production LLM inference usually degrades gradually rather than failing immediately. He identifies four key mechanisms: KV-cache fragmentation causing 60-80% memory loss, OOM on long contexts, head-of-line blocking in batches, and p50/p99 metric divergence. The article provides reproduction scripts and metrics to catch issues before incidents. It highlights that KV-cache often exceeds model weights in production. PagedAttention reduces fragmentation to less than 4% and boosts throughput 2-4x, but external fragmentation and eviction still cause issues over time. Continuous batching can cause head-of-line blocking when a long prefill request delays shorter ones; chunked prefill mitigates this. FP8 quantization of KV-cache nearly doubles effective memory capacity compared to FP16. Monitoring trends in cache usage and peak KV rather than waiting for OOM is advised.