Cloudflare: Stay discoverable in search while disallowing AI training

This explains a technical and policy approach to letting websites remain indexable for search engines while preventing their content from being harvested for AI model training. It frames the problem as one of mixed-use crawling - some bots only need to index pages for search, others want to ingest content into training datasets - and argues that website operators should be able to make a clear, enforceable distinction. The proposal centers on accountable crawlers that declare their intended use, are identifiable and auditable, and respect site-level directives so owners can allow discovery without consenting to training.

The substance outlines concrete controls and verification: leveraging and extending existing web controls (robots-style directives and HTTP signals) together with stronger crawler accountability such as authenticated identities, signed attestations or credentials, and monitoring to detect misuse. Cloudflare positions platform-level tooling to publish and enforce these signals, give site owners easy opt-out options for AI training while preserving SEO, and provide channels to block or penalize noncompliant crawlers. The approach emphasizes protecting creators’ rights and data consent without sacrificing discoverability, while noting that its effectiveness depends on broad adoption and enforceability across crawler operators.