LLM Inference: From KV-Cache to Production Deployment

The article by MLOps engineer Alexander Ryzhov from hh.ru discusses the challenges and solutions for deploying large language models (LLMs) on own servers. The key insight is that inference efficiency is dominated by GPU memory management. KV-cache shifts the decode stage from compute-bound to memory-bandwidth-bound. The author explains the evolution of attention mechanisms: MHA (2017), GQA (2023), MLA (2024) by DeepSeek, and DSA (2025) in DeepSeek v3.2, each reducing KV-cache size. Inference engines vLLM (open-source) uses PagedAttention to reduce memory fragmentation, while SGLang (open-source) uses RadixAttention for more flexible prefix caching. For hardware, SGLang is recommended for H100 and above, vLLM for Ampere and older. The sweet spot for one node (8x H100, 640 GB VRAM) is MoE models up to ~120B or dense models up to ~32B. Tensor Parallelism (TP) lowers latency, Data Parallelism (DP) maximizes throughput. Prefill/Decode disaggregation gives highest throughput but requires more engineering. Speculative decoding is not a panacea as it degrades under high concurrency. The article concludes with a checklist for production deployment.