Show HN: jevals – replacing LLM judges with typed Jev decisions

jevals is a Python library that replaces slow, expensive LLM judges with typed Jev-style decision models to evaluate agent traces, run guardrails, and gate agent behavior. It argues that traditional judge setups (multiple LLM calls, few-shot prompts, JSON parsing retries) are costly, slow, and highly non‑deterministic - LangChain comparisons showed GPT/Claude judges had 92x-913x more score variance than Jev. Jev instead takes structured state plus typed questions (yes/no, pick-one, rubric) and returns calibrated probabilities for each question in a single, parallel forward pass. That reduces cost and latency dramatically: examples show one request for eight metrics costing a few ten‑thousandths of a cent and ~0.3s, with pricing around $0.042 per million input tokens. Open-weight models (Kev, Laya) and a Vercel gateway are supported for local or low-latency use.

jevals provides an eval API centered on an Eval class with state(), questions(), and reduce() methods; multiple evals’ questions are merged into one request. Built-in checks include ToolChoice, UsedToolResult, Grounded, AnswerRelevancy, Completeness, IndirectInjection, PHI, and more. evaluate() works synchronously (aevaluate/acheck for async/gates) and accepts OpenAI chat format or other message types. Backends resolve from environment variables or can be passed explicitly (Jev via Typesafe/Vercel, Kev self-hosted, Laya local, or an LLM fallback); swapping backends is supported but requires recalibration of probabilities. Installation and examples are provided for running as offline metrics, production monitors, or in‑loop agent gates.