A large-scale experiment traced sensitive information through the layered network of AI apps, routing services, model providers, and hosting companies. Security researcher HD Moore sent nearly two million prompts containing unique, unused “canary” secrets, including AWS credentials, passwords, private repository references, and trackable URLs. Because each secret appeared in just one request, its later use could reveal where it had escaped. The experiment found that attackers compromised at least three AI hosting providers and used stolen AWS keys tens of thousands of times to probe cloud resources, seek additional credentials, and establish persistent access. Some stolen prompts had been rejected before reaching a model, indicating that attackers were copying requests at providers’ entry points.
Other leaks appeared to stem from business practices rather than intrusions: low-cost inference resellers logged prompts and passed them to people who tested embedded credentials. Providers also fetched secret URLs despite “no data collection” and Zero Data Retention settings. In one case, a lab’s scraper accessed URLs from more than half of tested prompts, including many routed through US-based resellers; a later attempt to clone private repositories also followed prompts sent to that lab’s models. Encryption in transit offers little protection because each intermediary decrypts prompts and may retain them. Moore recommends on-premises models where feasible, or direct contracts with a first-tier provider and carefully chosen upstream services, while stressing that neither guarantees privacy.
Summary generated by AI from the linked article. hn.today is not affiliated with Hacker News or Y Combinator.