AllenAI Open Instruct Tulu 3 Post-Training with SFT, DPO, RLVR, GRPO, and Verifier-Based Evaluation
AllenAI
This tutorial demonstrates an end-to-end post-training pipeline for a language model using AllenAI's Open Instruct framework, adapted to run on a 16 GB GPU. It covers Supervised Fine-Tuning (SFT), Direct Preference Optimization (DPO), and Reinforcement Learning with Verifiable Rewards (RLVR) using GRPO, with deterministic verifiers for evaluating mathematical answers.
The tutorial walks through building a compact instruction-tuned language model using AllenAI's Open Instruct framework. The pipeline includes three training stages: Supervised Fine-Tuning (SFT), Direct Preference Optimization (DPO), and Reinforcement Learning with Verifiable Rewards (RLVR) using GRPO, all adapted to fit a 16 GB runtime. The implementation leverages the Open Instruct repository, selectively loading functions like DPO loss, GRPO loss, and log-probability calculations. LoRA adapters are configured, and GSM8K data is prepared for each stage. Deterministic verifiers (GSM8K, Math, IFEval) are used to score generated mathematical answers. Distributed components such as vLLM, Ray actors, and DeepSpeed are replaced with lightweight Hugging Face and PyTorch implementations suitable for Colab. The model used is Qwen2.5-0.5B-Instruct.
- Abbreviations
- SFT = Supervised Fine-Tuning — обучение под наблюдением
- DPO = Direct Preference Optimization — оптимизация прямых предпочтений
- RLVR = Reinforcement Learning with Verifiable Rewards — обучение с подкреплением с проверяемыми вознаграждениями
- GRPO = Group Relative Policy Optimization — групповая относительная оптимизация политики
- LoRA = Low-Rank Adaptation — низкоранговая адаптация
- FP16 = Floating Point 16 — числа с плавающей точкой половинной точности
- BF16 = Brain Floating Point 16 — браузерные числа с плавающей точкой
- GPU = Graphics Processing Unit — графический процессор
- vLLM = Virtual Large Language Model (vLLM) — виртуальная большая языковая модель
- Colab = Google Colaboratory — облачный сервис Google
Source: MarkTechPost —
original