The Thinking Leakage in Reinforcement Learning for LLMs

Hybrid reasoning models that toggle between a THINK mode and a NO-THINK mode via a few prompt tokens are often trained with reinforcement learning (RL) in NO-THINK mode and report improved reasoning. This work demonstrates that much of that improvement is attributable to "thinking leakage": RL updates in NO-THINK mode partially reactivate the model’s existing THINK-mode circuitry rather than instilling fundamentally new reasoning strategies. Training Qwen3-1.7B, 4B, and 8B with GRPO in NO-THINK produced longer, more reflective outputs that the original model judged to resemble its own THINK responses, while internal activations shifted toward the THINK state as accuracy rose.

Mechanistic probes at the level of weights, activations, and attention support this account. Amplifying the RL update progressively elicits THINK-like behavior and approaches THINK-mode accuracy but does not exceed it. In Qwen3-8B, replacing a low-dimensional THINK-related activation subspace in the RL-trained model with the original-model values removes most of the RL gain, and the reverse swap transfers a large fraction of the gain to the original model. The RL-induced shift flows through attention heads that read the empty think block, and the GRPO advantage-weighted training signal favors and nudges the model toward THINK-like outputs. Thus observed NO-THINK gains largely reflect reactivation of preexisting reasoning circuits.