EleutherAI

Token Cramming: Ideal Text Compression Does Not Preserve Meaning – AIRI Study

Researchers at AIRI demonstrated that the token cramming method, which compresses up to 1568 tokens into a single embedding, does not guarantee meaning preservation: under greedy decoding, accuracy drops from 99.96% to 40%. A new method, progressive cramming, was proposed to achieve 100% reconstruction, but it was found that even a perfectly compressed embedding impairs the model's reasoning ability, and the cause lies in the early layers of the transformer.

Meta EleutherAI Hugging Face Google/DeepMind Alibaba/Qwen