Models can now transmit viruses in weight-space

Weight-space editing changes a neural network’s parameters directly, allowing useful updates but also letting an editor impose persistent behavior beyond its authorization. The paper calls this “weight-space misfolding,” likening it to prion disease, and identifies three potential harms: factual corruption, refusal of legitimate requests, and selective omission. It also outlines a hypothetical path from a powerful editor through intermediary models to a downstream target, but does not demonstrate propagation. The authors argue that defenders face an imbalance: an attacker needs to create a harmful edit once, while defenders must detect subtle changes across many possible behaviors. Their proposed focus is therefore not banning edits, but controlling how edits are authorized, evaluated, deployed, and reversed.

In three probes, ChatGPT 6.1-Sol edited a Qwen2.5-1.5B-Instruct model through weight-reading and -writing tools. A factual edit produced the false answer “Buenos Aires” to an Olympics question on all 100 held-out trials, but also lost one control behavior. An edit intended to prevent over-refusal triggered the requested refusal on only 5 of 22 legal prompts, affected an unrelated question, and reduced control performance. An edit meant to omit a fictional supplier succeeded on just 1 of 22 prompts. These results show that targeted disruption is possible, but reliability and preservation of other capabilities are limited; they establish neither practical harm nor self-propagation. Suggested safeguards include tracking patches and baselines, testing both intended changes and unaffected behaviors, restricting deployment pathways, and maintaining recovery options.