A Jev that can ask Opus for help gives Opus accuracy at 4x lower cost

The piece examines whether language models can audit identity resolution systems, which combine customer records into profiles representing real people. The key risk is over-merging: grouping records belonging to different people. Because real datasets lack a definitive answer key, teams usually inspect sampled profiles by hand, a slow process that can require more than 1,000 reviews to estimate over-merging rates precisely. The authors built a benchmark of 1,000 simulated identities using IHME’s pseudopeople data, adding realistic errors such as typos, nicknames, outdated addresses, and household information. Half the profiles represented one person and half contained multiple people; models were asked to identify how many people each profile described.

Frontier models led the standalone results: Claude Opus reached 99.0% accuracy, while GPT-6 Luna scored 97.3% and Jev 95.6% at their best thresholds. Luna and Jev were much faster and cheaper - about 0.15-0.17 seconds per identity and $0.05-$0.10 per 1,000, versus Opus’s 2.2 seconds and $3.86 per 1,000 - but decision models lost accuracy on harder cases. Jev’s confidence scores were better calibrated: its errors tended to carry low confidence, making it useful as a first-pass judge that escalates uncertain cases to Opus. This hybrid approach reportedly approaches Opus-level accuracy at more than four times lower cost. The findings come from a synthetic benchmark, not a test on live customer data.