HBM Not Enough, AI SSD Poised for Explosive Growth

Large language model inference is hitting two walls: capacity and I/O. Model parameters keep growing, while long contexts, multi-turn dialogues, and agent tasks inflate KV Cache; MoE models reduce active parameters per computation but require storing expert weights far exceeding VRAM or DRAM capacity. In cloud data centers, expensive HBM is occupied by weights and KV Cache, limiting GPU utilization; on edge and device, limited DRAM or unified memory restricts model size. As memory struggles to handle inference loads alone, SSDs are entering the real-time inference data path, becoming a formal storage tier after HBM, VRAM, and DRAM. This shift is visible in inference infrastructure: Moonshot AI's Mooncake uses a KV Cache-centric disaggregated architecture, pooling CPU, DRAM, and SSD as distributed cache, with Mooncake Store supporting multi-level caching with DRAM and SSD/NVMe; in 2026, NVIDIA introduced the CMX context memory storage platform in Vera Rubin infrastructure, adding an Ethernet-connected flash tier for KV Cache, with BlueField-4 managing NVMe SSDs. However, traditional SSDs are not suited for persistent inference tasks: they are designed for file and block storage, focusing on sequential bandwidth, random IOPS, and reliability, while LLM inference requires data supply with strict timing constraints. KV Cache is written in prefill and read repeatedly; MoE needs specific expert weights before each layer computation; a single missed read can stall the GPU, NPU, or CPU. Peak IOPS at high queue depths do not equal stable latency for low-queue-depth, fine-grained access. NAND Flash reads by page and erases by block, with FTL causing write amplification and tail latency; continuous KV Cache writes stress flash endurance. Traditional LBA layout ignores semantic relationships between layers, experts, and KV blocks, scattering related data and preventing predictable parallel reads. File systems, block devices, and NVMe software stacks add latency. Without address layout, cache partitioning, priority scheduling, async prefetch, and lifespan management oriented to model execution order, SSDs cannot participate stably in token generation like DRAM or VRAM. The market features two types of 'AI SSD': first, AI-load-enhanced enterprise SSDs, like InnoGrit's Dongting N3X series, using XL-Flash and SLC NAND for low latency and high endurance, targeting KV Cache offload and high-concurrency temp data, claiming one-third latency, triple write throughput, and 17-33x DWPD vs TLC SSDs; Huawei's OceanDisk series includes EX 560 for high random write performance, SP 560 for balanced performance, and LC 560 with up to 245TB capacity for training data and vector databases, with DiskBooster driver software enabling tiering with HBM and DDR. Second, inference-participating AI SSDs aim to change SSD operation logic, creating a virtualized NAND and DRAM/VRAM hierarchy through controller, firmware, and middleware. Phison's aiDAPTIV uses a cache SSD and middleware to stream model weights between VRAM and SSD, extending model size without adding GPUs, preserving KV Cache for reuse, with endurance up to 100 DWPD. Longsys's WM8500 SPU integrates compression, caching, and data scheduling, with iSA software making tiering and prefetch decisions, using 5nm process, lossless compression, HLC, and hybrid NAND scheduling. The Maxio-Infplane alliance (Maxio Tech and Infplane) combines Infplane's inference chip technologies like multi-level caching and prefetch with Maxio's NAND and SSD controller expertise, targeting MoE experts, KV Cache, and numerical precision, aiming for compatibility without modifying model files or frameworks, expanding verification across AI PCs, workstations, and edge devices. These routes differ in technical scope, with some enhancing I/O and others participating in runtime scheduling. Industry competition is just beginning, with validated products and commercialization attempts, and the future may not replace HBM, VRAM, or DRAM but re-tier capacity, performance, and cost, making SSD a key variable in token economics.