Can LLMs redesign themselves as they train?

The work explores whether an AI system can alter a neural network while it is training, rather than leaving architecture and training choices entirely to human designers. An agent examines the network’s internal state, identifies what it is failing to learn, and intervenes by changing its data, objective, update rule, or architecture. To do this, the agent develops its own tools, including a “phantom gradient” intended to measure unmet learning demands. It also writes small programs that become code layers inside the network. Some layers perform unusual operations, such as calculating gradients of hidden representations during a forward pass and using them to correct the network’s behavior.

Experiments illustrate this approach on algorithm-learning and reasoning tasks. On CLRS-TEXT, a transformer learned the input sizes in its training set but performed poorly on sizes between them; training twice as long did not solve the gap. Code layers added during training raised accuracy on those intermediate sizes from 33% to 79%. In a separate experiment, DeepSeek-V4.1-Flash served as an agent modifying itself: a code layer that read its hidden states improved its reported accuracy on AIME 2026 and HMMT 2026 from 64% to 71%. The central claim is that models can increasingly test interventions on themselves and reshape their own training and architecture, with people setting goals and constraints rather than designing every component.