hn.today

Training a model to identify AI-generated web content from structure alone

arxiv.org45 points9 comments
Screenshot of Training a model to identify AI-generated web content from structure alone

SlopShape trains a detector that distinguishes AI-generated commercial web content from human writing by analyzing structural signatures - how information is ordered, framed, evidenced and voiced - rather than relying on word-level cues. The study constructs a dataset of 2,250 pre-ChatGPT human blog posts from 268 company domains and 11,250 AI "mirrors" generated by five frontier models. A 214-feature annotation instrument is applied by an LLM and validated in a human gold-annotation session (human-human kappa 0.928, human-model 0.946). From 187 structural features alone, the classifier achieves 98.0 macro-F1 on held-out companies, with performance unchanged (98.1) when each AI post is reworded by its originating model.

The structural signal both characterizes and attributes content: AI-authored posts tend to share a tidy, self‑announcing shape, human posts occupy rarer structural configurations, and source attribution reaches 79.3% accuracy against a 16.7% chance baseline. Results replicate and amplify prior StoryScope findings for fiction, showing consistent direction and larger magnitude on commercial content. The author releases the full pipeline, instrument, prompts, code and aggregate artifacts to support replication and further study.

Read on arxiv.org9 comments on Hacker News

Summary generated by AI from the linked article. hn.today is not affiliated with Hacker News or Y Combinator.

More in Security

AI Exec: We May Have Pulled Off "The Largest Theft of Labor in Human History"

AI Exec: We May Have Pulled Off "The Largest Theft of Labor in Human History"

Documents from a lawsuit reveal that AI companies like OpenAI and Microsoft may have committed the largest theft of labor in history by training their models on copyrighted content without permission. Executives discussed concerns about AI's potential to destabilize the economy and the internet, highlighting unfiltered views on legal and ethical issues. (motherjones.com)

The daily digest

Today's best Hacker News stories, summarized and screenshotted, one email a day.