AI Can Read Neural Network Weights. I Wasn't Ready for That. – Arcturus Labs

Frontier language models can reportedly inspect neural-network weights as ordinary text, infer the behavior they encode, and even suggest changes without running a training loop. This seems surprising because weights are just numbers tokenized like other text, not a format the model can directly experience; the networks also lack semantic clues such as variable names. The author compares this with humans’ inability to understand a fruit fly’s brain merely by examining its wiring. A prompt supplied a model with the architecture and weights of a small game-playing network and asked it to identify the game.

Several models identified the network as a Pac-Man-style maze game. GPT-6 Astra reasoned from structured weight patterns: corridors shared by the player and moving entities, signals for collectible targets, inputs encoding previous positions, and differentiated outputs for movement directions. It inferred pellets and moving ghosts, while noting that weights alone could not establish exact reward rules or the presence of power pellets; it assigned 93% confidence. The author considers whether labs deliberately trained this skill, but argues it may instead show how far models generalize from program tracing and code reasoning. Even a small network had 569 variables and more than 17,000 weighted connections, yet models extracted an abstract description quickly. The demonstration challenges the author’s assumptions about how much information models can hold and how effectively they can reason in unfamiliar domains.