Jev Can't Be Calibrated

Jev is a "System One Model" that takes unstructured input and returns typed decisions from a predefined output set, attaching a probability to each choice. It shines because it works out of the box on classification tasks without requiring task-specific training data, making it a universal classifier where a fine-tuned BERT would need labeled examples. The central claim under scrutiny is that Jev's probabilities are calibrated; however, calibration is not purely a model property but a joint property of model and data distribution. Training Jev with RLCD can produce calibrated probabilities on TypeSafe’s training and evaluation distributions, but the same probabilities will be returned for different users even when their underlying data distributions differ, so calibration can break in deployment.

Empirical failures go beyond distribution shift: experiments and tweets show Jev assigning wildly incorrect probabilities (e.g., a fair coin at 0.92), and different primitives (Noul vs Choice) yield different calibration behavior, calling the meaning of "calibrated probabilities" into question. Practical advice is to treat Jev’s outputs as high-quality scores for ranking, not guaranteed probabilities. If true probabilities are required for thresholds, cost calculations, or model fusion, recalibrate on your production data - often just a few hundred labeled examples, using Platt scaling or similar, suffices - and always measure calibration before trusting numeric outputs.