Claude was faster and cheaper. We took the scenic route anyway

The study tests whether five small language models running locally can produce more reliable answers when their cited evidence is checked before a final answer is selected. Llama 3.2, Mistral, Gemma, Phi-3 and Qwen answered questions using retrieved passages; a separate hosted model, Claude Sonnet, judged whether each cited passage supported, contradicted or failed to address its answer. A fixed rule combined those checks with a majority vote, while answers, sources and judgments were saved separately. The checker assessed whether evidence supported a claim, not whether the evidence itself was true - a passage can support a false answer.

On 1,000 questions, the initial rule raised accuracy from 77.1% for majority voting to 85.0%. Among questions where models disagreed, it corrected 91 vote errors but introduced 12 errors, exceeding the pre-set safety limit for damaging answers. A revised rule - preserve the initial answer when it already has supporting evidence - was tested with a different set of models on the same questions. It corrected 73 errors and introduced three, passing the safety threshold; this does not establish performance on new questions. Sonnet choosing directly was more accurate on the first test set (93.9%) and cheaper than the checking method, but saved checks let researchers inspect decisions and rerun selection rules without further model calls. Removing or replacing cited passages erased most initial fixes, indicating that evidence mattered.