From GPT-2 to gpt-oss: Analyzing Architectural Advances and Comparison with Qwen3

OpenAI

Alibaba/Qwen

Meta

Google/DeepMind

AI21 Labs

Tencent

OpenAI released gpt-oss-120b and gpt-oss-20b, their first open-weight models since GPT-2. The architecture features Mixture-of-Experts, Grouped Query Attention, RoPE, SwiGLU, and MXFP4 optimization for local inference. Comparisons with GPT-2 and Qwen3 highlight advances in width vs depth trade-offs and attention sinks.

OpenAI released gpt-oss-120b and gpt-oss-20b, their first open-weight LLMs since GPT-2 in 2019. The models incorporate several architectural improvements over GPT-2: removing dropout, replacing absolute positional embeddings with RoPE, switching activation from GELU to SwiGLU, adding Mixture-of-Experts (MoE), and using Grouped Query Attention. MoE increases total parameters but activates only a subset per token, keeping inference efficient. GQA reduces key-value heads to lower memory usage. The models use MXFP4 optimization to fit on single GPUs (20B on 16GB consumer GPU, 120B on H100 80GB). Compared to Qwen3, gpt-oss explores width versus depth trade-offs. Attention bias and sinks are also discussed. The article also briefly mentions GPT-5 in context. Benchmarks show competitive performance, though detailed scores are not provided.

Сокращения: LLM = Large Language Model — большая языковая модель; MoE = Mixture of Experts — смесь экспертов; GQA = Grouped Query Attention — группированное внимание запросов; RoPE = Rotary Position Embedding — вращательное позиционное встраивание; GELU = Gaussian Error Linear Unit — гауссова линейная единица с ошибкой; GPU = Graphics Processing Unit — графический процессор; MXFP4 = Microscaling FP4 — микро-масштабирование FP4; KV = Key-Value — ключ-значение; GLU = Gated Linear Unit — линейный блок с затвором; SiLU = Sigmoid Linear Unit — сигмоидальная линейная единица; MHA = Multi-Head Attention — многоголовочное внимание

Source: Sebastian Raschka — original