Managing Reasoning Levels in Large Language Models
OpenAI
DeepSeek
Moonshot AI
The article explains how reasoning models like DeepSeek-R1 and GPT-5.6 learn to produce reasoning traces via reinforcement learning with verifiable rewards (RLVR). It details the training process, inference scaling, think tokens, and how reasoning effort modes (low/medium/high) are implemented through supervised fine-tuning and tokenizer switches.
Since OpenAI released o1, reasoning models that produce intermediate reasoning traces have become standard. DeepSeek-R1 popularized RLVR training, where a model is rewarded for correct answers on math and code tasks, without using the reasoning trace itself for updates. This led to "Aha" moments where models learn to self-correct. Kimi K1.5 and Tülu 3 also contributed to this approach, with R1-Zero showing pure RLVR can teach reasoning. Inference scaling, such as self-consistency via majority voting, further improves performance. Think tokens (<think></think>) are cosmetic markers to separate reasoning from answers, not essential for reasoning. Reasoning mode toggles (on/off) are implemented via supervised fine-tuning with think and no-think examples, reinforced during RL. Reasoning effort settings (low/medium/high) in models like GPT-5.6 likely control response length and accuracy, possibly via system prompts, as seen in OpenAI's gpt-oss models.
- Сокращения
- RLVR = Reinforcement Learning with Verifiable Rewards — обучение с подкреплением с проверяемыми наградами
- SFT = Supervised Fine-Tuning — обучение с учителем
- RL = Reinforcement Learning — обучение с подкреплением
- LLM = Large Language Model — большая языковая модель
Source: Sebastian Raschka —
original