Global-Batch Load Balance: Almost Free Lunch to Improve MoE LLM Training

The Mixture-of-Experts (MoE) architecture is a popular technique for scaling model parameters, but existing training frameworks like Megatron-core use micro-batch level load balancing loss, which can hurt performance when micro-batches contain homogeneous data (e.g., only code). Alibaba Qwen introduces global-batch balance loss, which synchronizes expert selection frequency across all parallel groups and calculates the loss globally. Experiments with three MoE configurations (3.4B/0.6B activated, 15B/2.54B activated, 43B/6.6B activated) and up to 400B tokens show that global-batch balance consistently outperforms micro-batch balance on various tasks. It also leads to significant domain specialization of experts, which is absent in micro-batch balanced models. Adding a small micro-batch balance loss (weight 0.01) on top of the global loss improves training speed without affecting effectiveness. The method is nearly free because the synchronization involves only a vector of size equal to the number of experts. The work is published in arXiv preprint 2501.11873.