Diffusion Language Models: How They Work, What Works, and What Raises Questions
Сбер
OpenAI
Microsoft
Diffusion language models (dLLMs) generate text in parallel, filling masks anywhere in the sequence. They offer advantages like parallel generation, text infilling, and robustness to reverse queries, but face issues like memory inefficiency and fixed-length generation. The article discusses recent research, including LLaDA, Plaid, and flow-matching methods, and explores adaptation of pretrained autoregressive models.
Diffusion language models (dLLMs) generate text in parallel, filling masks anywhere, unlike autoregressive LLMs which generate token-by-token from left to right. The masked diffusion approach, related to BERT, uses a forward process that replaces tokens with MASK and a reverse process that restores them. Advantages include parallel generation, text infilling, robustness to reverse queries (as shown by LLaDA), and better performance with limited data. However, dLLMs suffer from KV-cache inefficiency, independent token assumption, and fixed-length generation, requiring tricks for variable length. Evaluation often uses perplexity (GenPPL) but suffers from repetition collapse; diversity metrics are also needed. Adaptation of pretrained autoregressive models to dLLMs is a standard approach, with methods like DiffuGPT, DiffuLLaMA, Dream, and LLaDA, which was continued pretrained on Ling 2.0 to reach 16B and 100B parameters. The article also mentions the authors' own experiments and challenges.
- Abbreviations
- LLM = Large Language Model — большая языковая модель
- dLLM = diffusion Language Model — диффузионная языковая модель
- AR = autoregressive — авторегрессионный
- MDM = masked diffusion model — маскирующая диффузионная модель
- KV = Key-Value — кэш ключ-значение
- GenPPL = generative perplexity — генеративная перплексия
- MoE = Mixture of Experts — смесь экспертов
Source: Habr — хаб ИИ —
original