Calling the AI bluff: Adding "Do not guess" cut made-up fields from 71% to 20%

This reports a benchmark of web-extraction services that measures whether extractors invent field values when those fields are not present on a page. Extractors were given pairs of near-identical pages where one page contained a true value and the other contained a decoy or no value; honest behavior is to return the value when present and null when absent. Including a simple explicit instruction telling extractors to return null for missing fields and not to guess cut fabricated fields dramatically: across 16 tested models, fabricated outputs fell from 405 of 573 missing-field instances (70.7%) without the instruction to 116 of 574 (20.2%) with it. Some models performed very well with the instruction (e.g., Gemini 3.8 Flash, GLM 5.3), while others (notably Firecrawl) still copied decoys frequently.

A buyer agent can cheaply validate returned values by asking a small checker model whether the page actually supports each value. In this run GPT-6 Luna flagged 38 of 49 fabricated values and rejected none of 47 correct values; Jev flagged 23 of 49 and rejected none of 48. Checking all unique page/value pairs cost roughly half a cent to a cent per full benchmark run. Results are from a single run on synthetic test pages, paid APIs were exercised on free tiers, and email-contact traps and some previews were excluded; the benchmark currently covers only web extraction.