NVIDIA NeMo AutoModel Accelerates MoE Fine-Tuning 3.7x with Expert Parallelism and DeepEP

NVIDIA NeMo AutoModel is an open library within the NVIDIA NeMo framework for building custom generative AI models at scale. It builds on Transformers v5, adding Expert Parallelism (EP), DeepEP fused all-to-all dispatch, and TransformerEngine kernels, achieving 3.4-3.7x higher training throughput and 29-32% less GPU memory on fine-tuning MoE models compared to native Transformers v5, using the same from_pretrained() API with a single import line change. NeMoAutoModelForCausalLM subclasses AutoModelForCausalLM, enabling API compatibility with Hugging Face Transformers. The performance gains come from Expert Parallelism reducing memory pressure by distributing expert weights across GPUs, DeepEP fusing communication with computation, and TransformerEngine kernels accelerating core operations. Benchmarks include full fine-tuning of NVIDIA Nemotron 3 Ultra 550B A55B across 16 nodes (128 GPUs) with EP=64 achieving 815 TPS/GPU and 58.2 GiB peak memory, while Transformers v5 ran out of memory. Single-node benchmarks on 8x H100 80GB for Qwen3-30B-A3B showed 3.69x speedup (11,340 vs 3,075 TPS/GPU) and 29% memory reduction (48.1 vs 68.2 GiB), and for Nemotron 3 Nano 30B A3B showed 3.36x speedup (15,421 vs 4,583 TPS/GPU) and 32% memory reduction (42.5 vs 62.1 GiB). NeMo AutoModel ships hand-tuned implementations for popular MoE architectures like Qwen3, NVIDIA Nemotron, GPT-OSS, and DeepSeek V3, falling back to vanilla HF for others. It uses a balanced routing gate for benchmarks to emulate ideal operating point. Expert Parallelism is configured via distributed mesh, allowing EP and data parallelism to compose on the same devices (e.g., ep=8, dp=8).