Artificial Analysis Decision Index v1.0

Artificial Analysis Decision Index v1.0 benchmarks "decision models" - models that return a structured choice with a probability for each option in a single call - against four equally weighted evaluations: AA-Omniscience-Decision (knowledge and hallucination, 500 private questions with an "I don't know" option), HLE-MultiChoice (513 reasoning multiple-choice items), When2Call (NVIDIA tool-calling scenarios), and WildGuardTest (safety classification). The index reports accuracy, non-hallucination behavior, tool-calling performance, and classification, and compares decision models to language models that produce structured answers without per-option probabilities. Cost and latency are measured too, with decision models billed for input tokens only, enabling clear cost/time trade-offs versus LLMs.

Results show Jev 1.13 leading the Decision Index at 58, ahead of GLiDE and Microsoft-Decision-1 at 54; Jev excels on HLE-MultiChoice (21) and classification (WildGuardTest 94) but lags on AA-Omniscience accuracy (35) where Liquid AI's d1 reaches 41. Costs per million tasks span $17-$199 (Decider v1.1 27B cheapest at $17; Microsoft-Decision-1 $31; Jev $42; GLiDE $119) and median latencies 0.06-0.5s (Celeris-1 Decision fastest at 0.06s; Jev 0.13s). Decision models are systematically overconfident, especially on knowledge and reasoning items, while matching LLMs on classification and tool-calling. LLMs score higher on knowledge (Claude Haiku 5.5 Low 61; GPT-6 Luna Low 55) but are substantially slower and costlier (Claude ~1.4s/$184 per million; GPT-6 Luna ~1.5s/$131). Non-hallucination leaders: Microsoft-Decision-1 40%, Jev 35%, Mercury Decide 32%. Live leaderboards and full methodology are available for model submissions.