GSPO: Scalable Reinforcement Learning for Language Models

Alibaba Qwen introduces Group Sequence Policy Optimization (GSPO), a reinforcement learning algorithm designed to scale language model training. Unlike existing algorithms like GRPO, GSPO defines the importance ratio based on sequence likelihood and performs sequence-level clipping, rewarding, and optimization. This approach resolves severe instability issues observed in GRPO during long training, including irreversible model collapse. GSPO demonstrates significantly higher training efficiency, achieving better performance under the same training cost, and enables continuous performance improvements with increased compute. It also inherently stabilizes the training of large Mixture-of-Experts (MoE) models, eliminating the need for the Routing Replay strategy required by GRPO. Additionally, GSPO is more tolerant to precision discrepancies, making it infrastructure-friendly for scenarios like partial rollout and multi-turn RL. GSPO has been successfully applied to the large-scale RL training of the latest Qwen3 models, further unleashing the potential of RL scaling.