My Reddit AMA on the Law of Web Scraping

Eric Goldman, an Internet-law professor, opened a Reddit AMA on the law of web scraping with a clear disclaimer that nothing he said was legal advice and that he was focused on U.S. law. He summarized his deep experience dating to the 1990s and explained his ambivalent policy stance: he supports property owners’ rights to control resource use but also values broad, permissionless access to information because scraping fuels search, price comparison, AI training, and other socially beneficial services. He warned that ubiquitous age-authentication and anti-scraping tactics can turn the web into a permissioned environment, and he described the doctrinal confusion around key cases (e.g., Intel v. Hamidi, Register.com v. Verio, X v. Bright Data) that has left scraping “unteachable” in his classes.

Goldman lays out a practical framework for trespass-to-chattels analysis: identify the chattel (usually servers), whether the defendant “used” it (packets/requests), whether that use was unauthorized (and the contested effectiveness of TOS, C&Ds, robots.txt, IP blocks, registration walls), and whether the use caused legally cognizable harm. He emphasizes that scraping must also survive CFAA and state computer-crime laws (e.g., Cal. Penal Code 502), common-law trespass, copyright issues (including RAM copies and republication), DMCA 1201 complexities, and TOS-formation questions, concluding with the capsule judgment: “Scraping is ubiquitous but not clearly legal.”