Long-running coding agents often compress their conversation by turning messages into a summary and using that summary as future memory. The study identifies a security flaw in agents that first flatten role-tagged messages into plain text, using labels such as “User:”. Because the labels are not safely distinguished from content, an attacker can place text in files or other material the agent reads that imitates a user message. During compaction, a forged approval can then be preserved as if it were a genuine instruction. Tests found this attack succeeded 87-100% of the time across five flattening agents, even with context windows approaching 908,000 tokens, but 0% across four agents that retained structured message roles.
The researchers traced the flaw to consequential behavior, not just misleading summaries. Laundered approval led tool-running agents to execute attacker-controlled shell commands in a sandbox; in a live pi interface, the attack worked about half the time, compared with no successes in a clean control group. The same technique could steer an agent away from a file an attacker wanted hidden, potentially undermining code review of a planted backdoor. Since the damage occurs in harness code before summarization, prompt instructions cannot restore the lost provenance. A code-level provenance check caught 77% of attacks while incorrectly demoting 2% of legitimate summary content. The proposed direct fix is to escape role labels before flattening; preserving role-typed messages avoids the vulnerability altogether.
Summary generated by AI from the linked article. hn.today is not affiliated with Hacker News or Y Combinator.