hn.today

90% of US working hours are spent on tasks not covered well by AI benchmarks

mercor.com296 points274 comments
Screenshot of 90% of US working hours are spent on tasks not covered well by AI benchmarks

Mercor’s Benchmark Map measures how well publicly available AI benchmarks cover real U.S. work by mapping 29 selected benchmarks to O*NET’s 18,797 occupational tasks across 923 occupations. Using an LLM judge (validated on a subset) to tag prompts to O*NET tasks, a task counts as covered if mapped by at least five prompts. Combining those mappings with LLM-derived estimates of task time shares, the analysis finds that 90% of American working hours are spent on tasks with little or no benchmark coverage. Only 208 occupations have a task with at least five benchmark prompts, and coverage thresholds and selection rules drive the headline result.

Coverage is concentrated: benchmarks prioritize agentic, publicly available, economically realistic, unsaturated tasks and are capped per job family, which produces deep but narrow coverage biased toward AI-exposed activities and occupations with large wage bills (e.g., software engineering, financial analysis, and some legal tasks). Task-level importance and occupational wage-bill strongly predict inclusion; middle-income knowledge jobs and many job families remain undercovered. The report documents these patterns with task-level logistic regressions and caveats that the 29-benchmark sample biases estimates downward and will be expanded to fill gaps in understanding model capabilities across the economy.

Read on mercor.com274 comments on Hacker News

Summary generated by AI from the linked article. hn.today is not affiliated with Hacker News or Y Combinator.

More in AI

The daily digest

Today's best Hacker News stories, summarized and screenshotted, one email a day.