DoGBench is a benchmark for evaluating autonomous agents that write and maintain user-facing software documentation. It uses 292 tasks drawn from real open-source projects (205 requiring a documentation change, 87 requiring no change) and a primary, random stratified 117-item held-out split (82 needing updates, 35 needing no change). Each agent receives the repository state and a triggering signal (PR or reported gap) with merged documentation withheld. Task-specific rubrics (3,273 criteria, including 798 P0 criteria) score accuracy, completeness, placement, style, and conventions; 735 full agent trajectories from seven non-cloud agents are published. Cloud agents were instructed not to use the internet, with limited verification of compliance.
Results show that agents can produce polished prose but routinely fail on judgment and coverage. Agents misidentify when updates are needed, miss prerequisites or essential steps, contradict other pages, or fabricate details. In a 1,267-submission audit, 45.5% had task-completion gaps, 36.6% had technical inaccuracies, 32.5% omitted central concepts, and 6.1% contained fabricated content. Among the seven model-and-harness lanes, the top combined score on the held-out split was 47.3/100 (Qwen3.8 Max + OpenCode); a cloud agent tops the public leaderboard at 54.8. P0-clean delivery peaked at 39.0% (GPT-5.6 Sol + Codex). Scoring separates patch quality and correct abstention, combines them with a harmonic mean, and caps any patch with a critical failure at 60/100, underscoring that expert review remains necessary.
Summary generated by AI from the linked article. hn.today is not affiliated with Hacker News or Y Combinator.
AstaBrief is an open-source model designed for fast scientific report generation, capable of turning research questions and literature into cited reports. It reduces generation time by nearly three and a half times compared to proprietary models, making it a useful tool for researchers. (allenai.org)
Shrivu Shankar argues that SaaS companies are evolving into 'harnesses,' where core tasks are performed by AI agents rather than humans. This shift transforms organizational structure, with humans overseeing and reviewing AI-driven processes within the company. (blog.sshh.io)
Three AI agents - Meta’s Muse, Anthropic’s Claude, and OpenAI’s GPT - were tested on multilingual research tasks involving U.S. and Iranian data. The experiment focused on how language and context influence agent performance across reasoning, sourcing, and artifact creation. (royapakzad.substack.com)
An open-source project named LDRAW-NOVA allows users to generate Lego models using AI. The repository contains tools, examples, and documentation for creating Lego designs with artificial intelligence. (github.com)
DwarfStar 4 (ds4) is a local inference engine that runs large language and vision models on high-memory Mac, CUDA, and ROCm machines. It supports models like DeepSeek V4 and Qwen3.8, offering CLI, APIs, and persistent cache features for efficient local deployment. (dwarfstar.sh)
Rai is a CPU-only large language model inference engine written entirely in Rust. It aims to provide efficient and pure Rust solutions for LLM deployment without relying on GPUs. (github.com)
Today's best Hacker News stories, summarized and screenshotted, one email a day.