Using LLM for confidence scoring is a bad idea

The piece argues that asking a large language model for a probability or confidence score is unreliable because the model generates that number as text, not as a measured estimate based on evidence. Confidence, correctly understood, is a calibrated probability derived from available information and validated against outcomes over many cases; a trustworthy score must both reflect the information used and be checked against real outcomes. A human forecaster example illustrates calibration: if forecasts labeled 70% rain actually see rain 70% of the time, the forecasts are calibrated. An LLM’s numeric answer, however, is merely a plausible continuation of text shaped by tokens and prompts, so it changes with wording, previous sentences, or repeated asking and was never trained to match its written confidence to actual accuracy.

A reliable alternative is to build an external rule that converts observable signals into a confidence score and to continuously validate it. Practical steps are: decide which features to record (search hits, prior experience, answer agreement), start with a base rate from past outcomes, log every reply and its correctness, group past replies by feature patterns to estimate empirical accuracy for each group, and iterate the rule regularly by recalibrating against new records. Such a rule yields stable, evidence-linked, and improvable confidences; its weakness is that it depends entirely on diligently collected outcome data.