AI Measurement Science

AI Measurement Science presents a unified foundation for evaluating AI systems by treating evaluation as an inferential measurement problem: estimating latent constructs (reasoning, planning, safety, etc.) from observed responses rather than merely reporting dataset metrics. It argues that valid, actionable evaluation requires modeling assumptions, attention to construct validity, and explicit accounting for measurement error. The work develops probabilistic response models (Rasch and 2PL/3PL IRT, Bradley-Terry, factor models), estimation techniques (MLE, EM, Bayesian approaches), and diagnostics that link model assumptions to what can be inferred. It also provides tools for reliability (variance decomposition into person, item, interaction, residual), efficient design (Fisher information, CAT, D-optimal pools), and prediction-powered inference for cold-starts.

Beyond core methodology, it tackles when results generalize under distribution shift using causal models and conformal methods, and it analyzes strategic incentives once benchmarks influence development - Goodhart effects, signaling, gaming, and mechanism design to align stakeholders. Adversarial evaluation, synthetic-data scaling, and multimodal or multidimensional constructs receive applied treatment. The material includes interactive code and datasets, assumes basic statistics and ML knowledge, and closes with open challenges (multidimensional ability, temporal dynamics, agentic evaluation, scalable oversight, fairness) plus practical checklists for rigorous evaluation design.