Are Harnesses Worth the Cost?

A controlled experiment compared coding agents building the same backend-for-frontend service with and without a behavioral harness. The service had nontrivial requirements - HTTP semantics, schema validation, asynchronous Kafka publication of only the first product, timeout-to-202 semantics with polling links, specific retry and error codes, and interactions with a downstream Order API - and success required starting and passing all 178 external acceptance tests. The harness combined machine-readable guides (OpenAPI/AsyncAPI contracts, examples, and an API-development skill) with sensors (contract checks, API tests, CTRF and coverage reports) in an inner-loop development workflow, while an independent outer loop ran the acceptance suite. A frontier model, GPT-6 Sol, passed all 10 runs both with and without the harness, though the harness dramatically increased tokens, time, and cost. A cost-optimized model, GPT-6 Luna, succeeded in only 3 of 10 unharnessed runs but achieved 10 of 10 with the harness; harnessed Luna runs matched Sol’s reliability while costing 33% less per run ($0.273 vs $0.406).

The harness enforced deterministic, executable expectations rather than relying on the agent’s interpretation: examples were validated against specs, contract-driven mocks made downstream behaviors testable, and the agent invoked a test command to receive targeted, computational feedback for fixes. This separation of guides and sensors reduced the risk of an agent simultaneously mis-specifying implementation and tests. The outcome demonstrates that executable guidance and automated verification can convert a cheaper model into a reliably successful developer agent for a well-specified task, though results depend on the quality of the brief and supplied contracts.