DAPO: An Open-Source RL System from ByteDance Seed and Tsinghua Air

This release presents DAPO, an open-source large-scale reinforcement learning system for tuning LLMs that introduces the Decoupled Clip and Dynamic sAmpling Policy Optimization algorithm. It provides algorithmic details, training code, curated datasets, model weights (DAPO-Qwen-32B trained from Qwen2.5-32B), and full reproduction recipes. Using DAPO, the authors report state-of-the-art performance on AIME 2024 - 50 points with Qwen2.5-32B, matching or exceeding prior bests while using only half the training steps of the previous SoTA. Training diagnostics show longer, more stable responses, steady reward improvement, and entropy trends that balance exploration and exploitation.

The release includes a math-focused training set (DAPO-Math-17k), validation on AIME 2024, and two reproducible training scripts: a baseline without token-level policy-gradient loss and dynamic sampling (AIME 44) and the full DAPO pipeline (AIME 50), with wandb training records provided. Practical assets cover environment setup (conda, requirements), inference examples using vLLM and Ray Serve, recommended sampling settings (temperature 1.0, top_p 0.7, max_tokens 20480), and instructions for tensor-parallel deployment. Experiments were run on Volcano Engine and built on the verl RL framework; full evaluation/evaluation scripts and checkpoints are provided for researchers to reproduce and extend the results.