LLM Classification Is Feature Engineering

The piece argues that using large language models as end-to-end classifiers is the wrong framing and that LLM outputs should be treated as features for conventional ML models. LLMs produce useful signal but give hard, poorly calibrated labels, ignore structured covariates, and leave unclear how prompt content influences decisions. Wrapping an LLM verdict as a numeric feature in a logistic regression (p(y=1|x)=σ(α+β·LLM(x))) and fitting parameters on labeled data restores calibration, threshold control, the ability to include other covariates or reweight for different populations, and clearer attribution of the LLM’s contribution. Any downstream model (logistic regression, xgboost, neural nets, rules) is plug-and-play once features are extracted.

An empirical test on SemEval 2018 irony detection (4,618 tweets) uses Gemini-3.1 with a one-shot prompt and achieves TPR 0.965, FNR 0.035, Brier 0.259, F1 0.747. Fitting a logistic regression over the LLM verdict calibrates probabilities and reduces Brier to 0.175 with unchanged F1. Error analysis motivates richer features: log-probabilities, repeated runs, sub-verdicts, and a battery of linguistic subquestions (humor, plausibility, need for context, contrast, rhetorical form, etc.). The overall claim is practical and specific: LLMs excel at feature engineering, and modest labeled data plus standard supervised models produce more reliable, interpretable classifiers.