hn.today

Unlocking Stable On-Policy RL on TPU with Bitwise-Zero Train–Inference Mismatch

developers.googleblog.com3 points0 comments
Screenshot of Unlocking Stable On-Policy RL on TPU with Bitwise-Zero Train–Inference Mismatch

On-policy reinforcement learning for language models assumes that the policy used to generate tokens is exactly the one evaluated during training. Yet separate sampler and trainer execution paths can produce different token log-probabilities even with identical weights, because differing tensor shapes, kernels, differentiation, and distributed reduction orders change floating-point rounding. These small discrepancies can accumulate across long interactions and undermine stable learning. Google’s Zero-TIM implementation for JAX and TPU aims to eliminate that hidden off-policy gap: sampler decode, serving prefill, and trainer must return bitwise-identical probabilities for the same action and token history. Comparing all three paths also helps identify where discrepancies arise.

Zero-TIM aligns computation without merging serving and training into one engine. Its approach fixes within-shard accumulation order and precision, makes cross-shard addition order explicit, shares the action-scoring and log-softmax path, and reuses serving’s forward computation during training with custom gradient rules. In a 200-step FrozenLake test using Qwen3-8B on 64 TPU v5p chips, Zero-TIM recorded zero mean absolute sampler-trainer log-probability difference at every measured step and rising solve rates. Standard training and token-level truncated importance sampling retained discrepancies; the latter adjusts their effect on updates but does not remove them. Zero-TIM achieved higher observed solve rates in these single-seed runs. The modular implementation is part of Tunix, with support for additional model architectures under development.

Read on developers.googleblog.com0 comments on Hacker News

Summary generated by AI from the linked article. hn.today is not affiliated with Hacker News or Y Combinator.

More in AI

Mapping every neuron in Qwen3-8B

Mapping every neuron in Qwen3-8B

Researchers found individual neurons in Qwen3-8B that reliably steer its replies into distinct personas or styles, including pirate, German-speaking, and childlike voices. Nudging these neurons changed outputs across many questions without prompting or fine-tuning, revealing how specific behaviors can be localized and edited inside a language model. (fermionresearch.com)

ChatGPT for SNES

ChatGPT for SNES

ChatGPT for SNES runs a 2.2-million-parameter language model on the Super Nintendo’s Super FX 2 chip, with an optional Wikipedia fact book. The unofficial experiment can run in an emulator or on a flash-cart-equipped console, but its tiny model often gives incorrect answers. (bxfoundry.com)

The daily digest

Today's best Hacker News stories, summarized and screenshotted, one email a day.