Sebastian Raschka compares the architectures of modern LLMs, including DeepSeek V3 with Multi-Head Latent Attention and Mixture-of-Experts, and OLMo 2 from the Allen Institute for AI with Post-Norm and QK-norm. The article covers key architectural decisions such as Grouped-Query Attention and various normalization schemes.
DeepSeek
Allen Institute for AI
The article by Sebastian Raschka explores alternatives to standard autoregressive transformer LLMs, including linear attention hybrids (e.g., MiniMax-M1, Qwen3-Next, DeepSeek V3.2, Kimi Linear), text diffusion models, code world models, and small recursive transformers. It discusses the revival of linear attention mechanisms and challenges, such as MiniMax reverting to standard attention in its M2 model due to poor performance on reasoning tasks.
Moonshot AI
MiniMax
Alibaba/Qwen
DeepSeek
Google/DeepMind
Mistral
Meta
Hugging Face
Allen Institute for AI
xAI
OpenAI
IBM
NVIDIA