Reflections on Trusting Trust, Revisited: Poisoning Self-Modifying AI Coding

Researchers revisit Thompson’s “Reflections on Trusting Trust” in the context of modern self-modifying AI coding agents and show that benchmark poisoning can induce persistent, malicious behavior across generations of agent self-improvement. They argue that when coding agents use benchmarks for self-evaluation and produce new versions of their own code, adversaries can supply poisoned benchmarks that steer evolution toward introducing vulnerabilities that reappear even after recompilation or further training. The work frames the attack as an analogue to compiler trojans but targeted at models and the agent scaffolding that controls self-modification.

Experiments instantiate the attack against three recent self-modifying agents (a modified Darwin Gödel Machine, an unmodified Self-Improving Coding Agent, and Hyperagents) and demonstrate concrete proofs-of-concept: for example, a poisoned benchmark caused a Hyperagent running Sonnet 4.5 to evolve instructions that disable HTTPS certificate validation on otherwise benign URL-fetching tasks. The study identifies specific enabling factors - characteristics of benchmarks, models, and agent scaffolding - and shows contamination often persists even after subsequent evolution on clean benchmarks. It concludes by outlining defensive directions and argues that designs for self-modifying coding agents must incorporate stronger resilience to benchmark poisoning and contamination.