Breaking the open web won't stop the bad bots

Proposed federal and New York laws aimed at “stealth” web crawlers would require automated tools to disclose their identity and purpose, with penalties or subpoenas for those that do not. Lila Bailey and Tess Vartanian argue that these measures mistake anonymity for harm and could chill public-interest work without stopping abusive AI-related scraping. Large-scale automated collection can burden websites and threaten publishers’ revenue, but identifying a crawler does not limit its requests, compensate for reused content, or restore traffic lost to AI chatbots. Under the proposals, a disclosed crawler could still hit a site thousands of times a minute, while an anonymous journalist investigating corporate conduct could face legal risk.

The authors describe how scraping supports investigations, research, preservation, accessibility, public health, and cybersecurity. They cite a Pulitzer-winning Wall Street Journal investigation based on 31,000 financial disclosures, the Internet Archive’s record of more than a trillion webpages, disability audits of over a million sites, and government threat scanning. Anonymity can protect research from retaliation and prevent observed subjects from changing their behavior. Broad, vague restrictions may also deter these activities and burden the First Amendment right to receive information. The authors recommend addressing concrete harms - such as excessive requests, fraud, or prohibited data use - through existing privacy, copyright, and consumer-protection laws, alongside interoperable technical standards for managing machine access. Any new liability, they argue, should depend on demonstrable harmful conduct, not anonymity itself.