hn.today

How the Claude Code team thinks about verification

claude.dev240 points188 comments
Screenshot of How the Claude Code team thinks about verification

The central argument is that coding agents can produce more reliable work, and handle far more tasks in parallel, when they must show evidence that changes work. Verification is now relatively cheap because agents can create tests and run them repeatedly; in one internal structured-file-generation test, an independent rubric-based grader improved success by as much as 10 percentage points. Three test types are especially valuable: end-to-end tests expose failures in complete user journeys and provide actionable traces or screenshots; property tests generate many inputs to probe invariants, round trips, and other constraints; and differential tests compare old and new systems on identical inputs, making them useful for migrations. Tests should be fast and dependable, since flaky or slow checks encourage agents to ignore failures or run fewer tests.

The Claude Code team treats running the changed application as the strongest practical verification, using repository-specific skills to launch and inspect web, mobile, desktop, or backend systems. Agents attempt to fix failing checks, but escalate to a human after three unsuccessful rounds. Code owners define checks, while a 39-item reliability checklist selects relevant historical failure patterns based on the diff. Reviews build on earlier rounds and use guidance from the base branch, so a change cannot rewrite its own standards; agents also run tests, builds, and linters before requesting shared review. New commit gates begin in observe mode before enforcement, with escape routes to prevent agents getting stuck. Finally, the team evaluates its checks against past incidents, reverts, and stable changes, measuring whether reviews catch real bugs without raising too many false alarms.

Read on claude.dev188 comments on Hacker News

Summary generated by AI from the linked article. hn.today is not affiliated with Hacker News or Y Combinator.

More in AI

Human.md

Human.md

Human.md argues that people working alongside increasingly capable AI should bring intentional rule-breaking, unexpected inspiration, empathy, emotional resonance, and delight. It urges creators to experiment and ship many prototypes rather than rely on polished opinions or merely meet a brief. (human-md.ai)

The daily digest

Today's best Hacker News stories, summarized and screenshotted, one email a day.