Why Neural Networks Eat Tokens: What Happens in Long Chats and How to Cut Costs

The article discusses why AI models consume tokens excessively in long chats and how to reduce costs. It cites Reddit anecdotes: one user burned through 1.156 billion input tokens with Claude, another forgot to stop a Claude Code agent that checked pull requests every 30 minutes for 26 hours, totaling 46 runs and costing roughly $6000. The technical reason for high costs is that every new request includes the entire accumulated context, which can grow to 800k tokens. Anthropic warns in its docs that costs rise with context size and recommends clearing history or using compaction. Research by Chroma ('Context Rot') and a 2026 study ('Diagnosing and Mitigating Context Rot') show that performance degrades with longer inputs, even before the context window limit, especially with distracting information. Prompt caching only reduces costs for repeated prefixes, not the problems of unnecessary context. For AI agents, costs are hidden because they perform many tool operations autonomously. Practical tips include splitting independent tasks, using /clear and /compact in Claude Code, giving only relevant document sections, caching stable instructions, and picking the right model tier (Sonnet for coding, Opus for complex reasoning, Haiku for simple subagent tasks). The author also promotes SYNTX.AI's LLM Studio, which offers over 100 AI models and the ability to switch models within a dialog, with promo code CTRLAI for 20% discount. The article concludes that prompt engineering must now include discarding unnecessary information.