Does an open-weight decision model beat a hosted one? Jev vs. Laya

This compares a hosted typed-decision service, Jev, with a downloadable open-weight model, Laya, for making bounded agent decisions locally. Laya is a 421M-parameter model with a ModernBERT-large encoder and a decision head that scores option markers in a single bidirectional pass, returning structured choices, scores, or yes/no probabilities so no text parsing is needed. The local setup runs on an M3 Ultra Mac Studio, exposed via a FastAPI wrapper that accepts Jev-style requests, enforces strict budget and capability checks, refuses downloads during inference, and treats model outputs as recommendations only. Laya’s training used synthetic workflows and rewards for honest probabilities, but it remains overconfident without domain calibration; typed answers can be structurally correct yet factually wrong, so deterministic checks, abstain options, and fallbacks are mandatory. Jev wasn’t selected because the requirement was routine local decisions with data and model control.

A frozen replay of 40 held-out routing decisions compared the deterministic quality-and-latency router to Laya under identical eligibility rules: the deterministic router yielded 33/40 acceptable decisions (82.5%), Laya produced 37/40 (92.5%), with Laya directly selecting a model 15 times, the gateway falling back 17 times, and blocking 8. Latency was a weakness: routing p95 was 143.5 ms on one run and 466.7 ms on a busy host, exceeding a 300 ms timeout and prompting fallbacks. Results apply to this deployment and hardware; no live Jev-vs-Laya side-by-side was run, and Laya cannot compensate when candidate models themselves fail.