AI21 Labs

From GPT-2 to gpt-oss: Analyzing Architectural Advances and Comparison with Qwen3

OpenAI released gpt-oss-120b and gpt-oss-20b, their first open-weight models since GPT-2. The architecture features Mixture-of-Experts, Grouped Query Attention, RoPE, SwiGLU, and MXFP4 optimization for local inference. Comparisons with GPT-2 and Qwen3 highlight advances in width vs depth trade-offs and attention sinks.

OpenAI Alibaba/Qwen Meta Google/DeepMind AI21 Labs Tencent