Can a Linear Program Do Weight-Space Programming?

Astra’s weight-space edit addressed repetitive “thinking loops” in Qwen3.5-0.8B, which often consumed its token budget without answering. The method identified a direction in the model’s residual stream associated with looping, then altered the output matrices of layers 15 and 19 so their contributions pushed against that direction. This is a rank-one weight change: it modifies how selected MLP columns contribute along one shared vector. Astra tested different strengths and columns, ultimately editing 64 columns per layer. The change improved math and instruction-following scores and reduced answers hitting the token limit. A model that rarely looped did not gain math ability, suggesting the edit removes an obstacle rather than creating a missing capability.

The central claim is that choosing the columns and edit strengths can be formulated as a linear program. By constraining the edit to push loop-associated activations away from the loop direction while minimally affecting productive activations, and minimizing the L1 norm, the solver finds a sparse change. A GPU implementation solved the problem in about a minute, replacing hours of trial and error. Using Astra’s direction, the LP matched its benchmark results while changing roughly one-eighth as many weights; a direction derived automatically from model traces achieved most of the benefit. The experiments also found that a nearly constant push along the direction could reproduce much of the improvement as an inference-time steering vector. The findings are limited to one model and loop removal; selecting layers automatically and applying the approach to other behaviors remain open questions.