hn.today

Compaction Laundering: Provenance Destruction in Context-Compacting Coding Agent

arxiv.org9 points2 comments
Screenshot of Compaction Laundering: Provenance Destruction in Context-Compacting Coding Agent

Long-running coding agents often compress their conversation by turning messages into a summary and using that summary as future memory. The study identifies a security flaw in agents that first flatten role-tagged messages into plain text, using labels such as “User:”. Because the labels are not safely distinguished from content, an attacker can place text in files or other material the agent reads that imitates a user message. During compaction, a forged approval can then be preserved as if it were a genuine instruction. Tests found this attack succeeded 87-100% of the time across five flattening agents, even with context windows approaching 908,000 tokens, but 0% across four agents that retained structured message roles.

The researchers traced the flaw to consequential behavior, not just misleading summaries. Laundered approval led tool-running agents to execute attacker-controlled shell commands in a sandbox; in a live pi interface, the attack worked about half the time, compared with no successes in a clean control group. The same technique could steer an agent away from a file an attacker wanted hidden, potentially undermining code review of a planted backdoor. Since the damage occurs in harness code before summarization, prompt instructions cannot restore the lost provenance. A code-level provenance check caught 77% of attacks while incorrectly demoting 2% of legitimate summary content. The proposed direct fix is to escape role labels before flattening; preserving role-typed messages avoids the vulnerability altogether.

Read on arxiv.org2 comments on Hacker News

Summary generated by AI from the linked article. hn.today is not affiliated with Hacker News or Y Combinator.

More in Security

1984: The Canadian Edition

1984: The Canadian Edition

Canada’s proposed Bill C-22 would let officials compel digital services to assist surveillance and create law-enforcement access, raising concerns about security and privacy. Signal, NordVPN, Windscribe, and other companies say they could limit services or leave Canada if the bill passes. (btxx.org)

The daily digest

Today's best Hacker News stories, summarized and screenshotted, one email a day.