Token Cramming: Ideal Text Compression Does Not Preserve Meaning – AIRI Study

AIRI researchers studied the token cramming method, which compresses up to 1,568 tokens into a single embedding of a frozen LLM with reconstruction accuracy >99%. However, when transitioning from teacher forcing to greedy decoding, accuracy drops to 40%, as errors concentrate on the first tokens. The authors proposed progressive cramming, which guarantees 100% reconstruction by incrementally adding tokens one by one. Using this method, they measured the compression capacity of various models: for example, Llama-3.1-8B compresses 1,438 tokens, and with low-dimensional projection, up to 1,697. Studying optimization trajectories showed they lie in a low-dimensional subspace (30–100 components). The main result: even a perfectly compressed embedding, when fed together with the original prefix, reduces accuracy on HellaSwag (from 61.9% to 40.8% for Llama) and drops to zero on MMLU. The reason lies in the early transformer layers: attention knockout revealed that interactions in the first layers are necessary for reconstruction but hinder understanding. Compression capacity monotonically increases with model depth and width. It was also found that the difficulty of compressing a token is predicted by its surprisal.