LLMs are not superhuman at reading and writing neural network weights

A recent study presents GPT-6 Astra as able to infer what games a small neural network learned and to improve a language model by editing its weights, capabilities that could appear to enable recursive self-improvement. The counterargument is that these results do not show superhuman AI research. In the game task, the network’s inputs correspond to cells in an 8×8 grid, and its input channels are labeled for features such as the agent and moving objects. Measuring the strength of first-layer connections across each grid cell produces heatmaps that reveal recognizable layouts: a player at the bottom for Breakout or the maze for Pac-Man. Astra’s ability to calculate these patterns without running code is impressive mental arithmetic, but a skilled researcher could interpret them too.

The stronger demonstration modifies a 0.8-billion-parameter Qwen model that failed at long mathematics tasks because it began repeating itself after 32,000 tokens - not because it lacked mathematical ability generally. Astra generated four pairs of looping and non-looping reasoning examples, identified weights associated with the looping behavior, reduced their influence, and tested changes until the repetitions stopped. This resembles established human techniques for steering models, rather than a radically new research method. Direct weight editing is compared to changing a compiled binary: useful in some cases, but less effective than working with source code. Likewise, the more consequential role for intelligent LLMs in model improvement is helping researchers improve training, not bypassing it through magical weight manipulation.