Jev vs. LLMs on 770 "Am I the Asshole?" posts

A benchmark compared a fast judgment model (Jev, TypeSafe’s System One) against Sonnet 5, GPT-5 nano, and two local open-weight models on 770 r/AmItheAsshole posts from 2025. Each model returned probabilities for four verdicts (YTA, NTA, ESH, NAH) and was scored primarily by weighted Brier score (lower = better), plus weighted top-1 accuracy, macro recall, median call latency, and cost per 1,000 posts. The target label was Reddit’s official verdict flair (the top-voted comment), so “correct” means matching that community verdict; baseline guessing by Reddit’s base rates (always NTA) is 74% accuracy. The benchmark also explored different Jev prompt formats (direct multiple choice vs chains of yes/no questions) and produced confidence intervals via resampling.

Unadjusted results put Sonnet 5 first (weighted Brier 0.344) and Jev direct second (0.369). A small post-hoc logistic adjustment trained on 300 older posts lowered Jev’s score to 0.337, but that adjustment was not a like-for-like retest and was chosen after seeing initial outcomes. A fold-adjusted check across models narrowed the gap (Sonnet 0.332 vs Jev 0.339), leaving the lead inconclusive. Jev’s strength is speed and cost: median call 0.39 s and ~$0.037 per 1,000 posts (6.3× faster and 62× cheaper than Sonnet), not the 40-200× speedups sometimes claimed. Only Jev and Sonnet clearly beat base-rate guessing; most models missed the rarer ESH/NAH verdicts, and many comparative findings are sensitive to sampling noise and the post-hoc adjustments.