Livenerf: Has Opus 5.5 been nerfed yet?

livenerf is a compact, append-only benchmark designed to answer a single operational question: does a frontier model degrade after launch? It runs a frozen, pre-registered panel of questions against Claude Opus 5.5 (launch 2026-09-22) once per day for 30 days, using a hermetic claude -p harness with a pinned CLI and deterministic prompts/graders so every non-random variable is fixed. Built on the Inspect framework and following Anthropic’s error-bar guidance, the schedule uses days 1-10 as baseline and two subsequent 10-day windows to detect drift. The primary metric is the paired per-item score difference (clustered SEs) and the secondary signal is output token count, since lower effort shows up in tokens before accuracy.

Progress and early findings are concrete: as of 2026-09-29 six baseline days were collected; the calibrated panel contains 78 “sometimes right” questions selected from thousands, with Opus 5.5 scoring ~93% first-try and 97% of items being always-right or always-wrong. Calibration revealed selection bias (pass rates rose from 54.7% to 62.0% on fresh samples) and power calculations reflect that. Sensitivity: one daily run detects ~7.5 accuracy points per 10-day window (~3.6% weekly). Validation shows reduced model effort cuts tokens massively and lowers accuracy (effort-low: −62% tokens, −8.3 ±4.5 points; medium: −26% tokens, −4.2 ±3.9). A swap from Opus 5 to 5.5 was indistinguishable at 99% (−3.8 ±6.3 points, −23% tokens). Audit flagged some bad/ambiguous items and safety-classifier rejections are excluded; the pipeline and pre-registration enforce reproducible detection rather than anecdote.