The Illusion of Thinking: Models Don't Actually Reason

The study tests whether large reasoning models genuinely gain reliable problem-solving ability from generating long “thinking” traces. Rather than relying on familiar math and coding benchmarks, which may be contaminated and reveal little about how answers are reached, the researchers use controllable puzzle environments. These preserve the same basic logic while allowing problem complexity to increase in a measured way, so they can compare final accuracy and the reasoning traces themselves. They also compare reasoning models with standard language models using the same amount of inference compute.

Results show three performance zones: standard models do better on easy tasks, reasoning models have an advantage at moderate complexity, and both types fail almost completely on sufficiently hard tasks. Reasoning models initially spend more effort as puzzles grow harder, but past a threshold their reasoning effort falls - even when they still have token budget - while accuracy collapses. The traces also reveal difficulties with exact computation: models do not reliably apply explicit algorithms and can reason inconsistently as problem scale changes. The findings challenge the idea that longer visible reasoning necessarily reflects robust, scalable reasoning. Instead, these models offer gains over standard models in a limited range of difficulty, with clear limits at greater complexity.