On-policy reinforcement learning for language models assumes that the policy used to generate tokens is exactly the one evaluated during training. Yet separate sampler and trainer execution paths can produce different token log-probabilities even with identical weights, because differing tensor shapes, kernels, differentiation, and distributed reduction orders change floating-point rounding. These small discrepancies can accumulate across long interactions and undermine stable learning. Google’s Zero-TIM implementation for JAX and TPU aims to eliminate that hidden off-policy gap: sampler decode, serving prefill, and trainer must return bitwise-identical probabilities for the same action and token history. Comparing all three paths also helps identify where discrepancies arise.
Zero-TIM aligns computation without merging serving and training into one engine. Its approach fixes within-shard accumulation order and precision, makes cross-shard addition order explicit, shares the action-scoring and log-softmax path, and reuses serving’s forward computation during training with custom gradient rules. In a 200-step FrozenLake test using Qwen3-8B on 64 TPU v5p chips, Zero-TIM recorded zero mean absolute sampler-trainer log-probability difference at every measured step and rising solve rates. Standard training and token-level truncated importance sampling retained discrepancies; the latter adjusts their effect on updates but does not remove them. Zero-TIM achieved higher observed solve rates in these single-seed runs. The modular implementation is part of Tunix, with support for additional model architectures under development.
Summary generated by AI from the linked article. hn.today is not affiliated with Hacker News or Y Combinator.