3 Bits Instead of 16: Breaking Down TurboQuant and When to Actually Enable It
Google/DeepMind
Mistral
Google DeepMind
Alibaba/Qwen
NVIDIA
Google's TurboQuant algorithm compresses the KV-cache of language models 5-6x without retraining or calibration, causing panic among memory manufacturers. Despite hype about '8x speedup' and 'zero loss', the real benefit is memory savings, enabling longer contexts on the same hardware. The article reviews how it works, its limitations, and compares it with alternatives like KVTC, RotorQuant, and KIVI.
In March 2026, Google published a post about TurboQuant, a memory compression algorithm for language models, which caused a market panic as memory manufacturer stocks lost nearly $90 billion in combined capitalization over a week. The algorithm addresses the KV-cache problem: as LLMs generate text, they store intermediate computations for all previous tokens, which grows linearly with context length and can consume tens of gigabytes. For example, the KV-cache for Llama 3.1 70B at 128K context occupies about 40 GB in BF16, making it impossible to fit on a single H100 GPU. TurboQuant compresses the KV-cache 5-6x by applying a random rotation to vectors before quantization, followed by an optimal Lloyd-Max quantizer, all without calibration or fine-tuning. This data-oblivious approach works on any transformer model. Google claims 'up to 8x speedup on H100 GPUs', but community analysis shows that is measured against FP32; real speedup is about 4x for attention only, and memory savings are the true benefit. Benchmarks on small models (up to 8B) show zero accuracy loss on LongBench and Needle-in-a-Haystack, but no data for 70B+ models, and community tests on a 104B model show only a 3.6% perplexity increase. Comparisons with alternatives like KVTC (up to 20x compression but requires calibration), RotorQuant (faster rotation but lower quality), and KIVI (16x compression, integrated into Hugging Face) provide a comprehensive picture. The lack of official code is mitigated by over 15 community implementations.
- Abbreviations
- KV = Key-Value — ключ-значение
- BF16 = Brain Floating Point 16 — 16-битный формат с плавающей точкой
- GQA = Grouped Query Attention — группированное внимание запросов
- RAG = Retrieval-Augmented Generation — генерация с дополнением поиском
- VRAM = Video Random Access Memory — видеопамять
- GPU = Graphics Processing Unit — графический процессор
- LLM = Large Language Model — большая языковая модель
- MLA = Multi-head Latent Attention — многоголовое латентное внимание
- ICLR = International Conference on Learning Representations — международная конференция по представлениям обучения
- QJL = Quantized Johnson-Lindenstrauss — квантованное преобразование Джонсона-Линденштрауса
- FP16 = Floating Point 16 — 16-битный формат с плавающей точкой
- FP32 = Floating Point 32 — 32-битный формат с плавающей точкой
- PPL = Perplexity — перплексия
- MMLU = Massive Multitask Language Understanding — массовое многоязычное понимание
- PCA = Principal Component Analysis — метод главных компонент
Source: Habr — хаб ИИ —
original