Tiered KV Cache for Large LLMs on Amazon SageMaker HyperPod with Curvine

Running large language model (LLM) inference at scale typically forces a KV cache trade-off: either paying for oversized GPU instances to accommodate a growing KV cache, or accepting slow time-to-first-token (TTFT) as identical prompts get recomputed on every request. For teams deploying a broad catalog of publicly available foundation models (FMs), such as Qwen, Llama, DeepSeek, and others, across per-business-line endpoints, Retrieval Augmented Generation (RAG) pipelines, or multi-turn dialogue applications, this trade-off translates directly into higher infrastructure cost and degraded user experience. The root cause is that during generation, vLLM stores attention keys and values for every token in a KV cache, and prefix caching reuses that cache across requests with shared leading tokens. On cost-efficient instances like ml.g6e.4xlarge (48 GB per GPU), memory left for prefix caching is limited, and cache hit rates drop on long prompts, identical system prompts get re-prefilled on every request, and horizontally scaled vLLM replicas each maintain isolated caches. The solution builds a three-tier cache hierarchy: L0 (GPU prefix cache), L1 (CPU memory offload), and L2 (shared distributed NVMe pool via Curvine). L0 is vLLM's native paged-attention layer, L1 uses LMCache to offload evicted GPU blocks to host DRAM, and L2 pools local NVMe drives into a shared namespace via Curvine, mounted as a ReadWriteMany PVC into every inference Pod. HyperPod's Intelligent Routing directs requests to replicas most likely to have relevant KV blocks, with strategies including prefix-aware, kv-aware, and round-robin. The net effect: only on a complete miss does the system re-prefill from scratch, substantially reducing TTFT for workloads with moderate-to-high prompt overlap. On a test deployment, this achieved up to 100% cross-Pod cache hit rate, up to 2.7x TTFT improvement, and cross-node L2 read latency of about 56 ms for a ~1,900-token prompt. With this architecture, workloads that previously required P5 instances can run on lower-cost G6e instances, reducing per-endpoint cost. The implementation requires SageMaker HyperPod with Tiered Storage enabled, an EKS cluster, and installation of the Inference Operator and Curvine. The post walks through five stages: enabling Tiered Storage, installing the Inference Operator and dependencies, installing Curvine, patching the Inference Operator for filesystem-backed L2, and benchmarking.