Build your own decision model

Describes a practical pattern for "system one" decision models that choose from a fixed set of answers in a single pass by constraining the model’s output vocabulary and selecting the highest-probability option. Provides a concrete implementation using Hugging Face transformers and Qwen3-1.7B: map each option (A-E) to its first token id, build a prompt with enumerated choices, run the model once to get logits, mask to only the option token ids, softmax those logits and pick argmax. A running example ("What color is the sky?") shows the model predicting B with probability 0.9988, demonstrating how constrained decoding yields a one-shot multiple-choice predictor.

Evaluates the approach on a CommonsenseQA holdout: baseline accuracy 725/1221 = 0.5938 and macro F1 0.5844; a quick fine-tune improves accuracy to 762/1221 = 0.6241 and macro F1 0.6234. Highlights a calibration problem: raw output probabilities are overconfident (predictions in the 0.9-1.0 bin are only ~70% correct). Applies temperature scaling (fitted temperature ≈ 3.7973) to flatten probabilities and align confidence with observed accuracy, which substantially improves bin-wise calibration (high-confidence bin accuracy rises to ~95%). Includes a GitHub repo with scripts to build datasets, evaluate, fine-tune and calibrate similar decision models.