hn.today

AI Crawler Index

mojodojo.io17 points1 comments
Screenshot of AI Crawler Index

A live index of crawler traffic to mojodojo.io shows which automated agents visit a real website, how often, and whether they’re verifiably operated by their claimed owners. Over the most recent 30 days, AI-related crawlers accounted for 24.2% of bot hits (14,037 of 58,057), nearly matching search engines at 25.9% and producing 1,034 live page fetches on behalf of users. The dataset excludes spoofed hits by checking visitor IPs against published ranges or forward-confirmed reverse DNS; 535 fake visits were removed and 11.5% of verifiable bot claims came from the wrong IP. Daily hits fluctuated with a peak of 6,724 on 8 Oct 2026.

The index breaks crawlers into AI training (17%), AI search (5.4%), AI user fetch (1.8%), SEO tools (23.2%), social previews (21.9%), archive (0.9%) and other bots (3.8%). A leaderboard lists top visitors: FacebookBot (10,350 hits), AhrefsBot (8,519), Bingbot (5,860), Googlebot (4,993), ClaudeBot (3,479) and Amazonbot (2,969), with metrics for hits/day, pages visited and hits per page. Methodology notes that tracking began June 6, 2026, with IP-based verification starting 7 Oct 2026; numbers are a detailed case study for one site rather than a global census.

Read on mojodojo.io1 comments on Hacker News

Summary generated by AI from the linked article. hn.today is not affiliated with Hacker News or Y Combinator.

More in Security

The daily digest

Today's best Hacker News stories, summarized and screenshotted, one email a day.