Can a Linear Program Do Weight-Space Programming?

The study examines whether a linear program can replace an AI agent’s trial-and-error search when editing a language model’s weights. Astra had reduced repetitive “thinking loops” in Qwen3.5-0.8B by identifying a direction in the model’s residual stream associated with looping, then changing selected MLP weights to push activations against that direction. The edit improved MATH-500 accuracy from 7.9% to 48.1% and reduced answers hitting the token limit from 93% to 33%. Because the edit changes only how strongly individual weight columns contribute along one shared direction, the author formulates the choice of columns and edit sizes as an L1-minimizing linear program, constrained to push loop-like tokens while minimally affecting productive reasoning.

Using Astra’s direction, the LP matched Astra’s benchmark performance while changing about 16,000 weights instead of 130,000; it also found useful columns different from Astra’s, suggesting multiple sparse edits can produce the same effect. A direction estimated automatically from repetition patterns achieved about 93% of the improvement. The LP’s search took roughly a minute on a GPU, rather than Astra’s eight hours of weight selection. Further tests found that a fixed steering vector added during inference nearly reproduced the benefit, indicating the successful edit acted largely as a constant push. The findings concern one model and one behavior; choosing the right layers and direction without Astra remains unresolved, and the method has yet to be tested on behaviors such as refusal or sycophancy.