We Tested Jev on 100 Agent Tool Calls

Researchers evaluated Jev on 100 real agent tool calls from production Claude Code sessions to see whether a fast, structured classifier can beat a simple constant baseline when labeling runtime tool calls for Information Flow Control. Each call required four labels (delta_audience, delta_trust, requires_audience, requires_trusted), yielding 400 decisions; 337 decisions where three independent judge families (Claude Opus, GLM 5.3 Flash, Gemini 3.8 Flash) agreed were reported. Benchmarked models included Jev, Sonnet 5, SemIf, Laya, Bespoke-Nimble-9B and a Qwen3 reranker. A majority-constant baseline scores 79%. Sonnet led at 98% zero-shot; Jev scored 93% zero-shot and 95% with few-shot prompting; Jev’s refusal recall for the nine dangerous calls was 78% (7/9) versus Sonnet’s 44% (4/9) and Laya’s 100% recall earned by flagging everything (very low precision). Self-grading biased scores upward by 2-3 points; models should not grade their own outputs.

Evaluation surfaced concrete failure modes and benchmark gaps: many trivial local calls (79%) create a “constant trap,” opaque IDs provide no semantic signal, and outbound queries can turn a read into an exfiltration risk. Small open decoders collapsed under option-position priors and benefited greatly from in-context examples; encoder-style models like Laya did not. Jev’s labels were mostly stable across repeats (394-398/400 identical) but probabilities drifted and order permutation changed ~4% of multi-way decisions, warning that tight probability thresholds and option ordering affect safety-critical routing. The experiment shows Jev is promising for confidence-based abstention, but specification clarity and stress-testing on edge cases are essential before trusting classifiers at security boundaries.