Experiments comparing GPT-6 Astra with GPT-5.6 Sol find that Astra both solves coding tasks more reliably and uses testing and verification methods more effectively. On a Zstd implementation from its specification, Astra achieved full correctness on hidden tests in over 99% of medium-effort runs for about $1, compared with Sol’s 63% at $3.50. Astra’s fuzzing also generated structured inputs that exercised meaningful code paths, though Zstd was too easy to reveal whether this improved correctness. On an IMAP task, default instructions remained competitive, while some specialized methods improved results at higher cost. A harder Pandoc implementation, tested across 25 runs per condition and roughly 14 turns on average, never reached full correctness; the evaluation cost about $90,000 at standard pricing.
Pandoc results suggest several methods may outperform default testing, but small samples, code-generation path dependence, and the use of just one large task make rankings uncertain. Differential testing stood out for prompting agents to examine boundaries and invariants; TDD again performed poorly, while other approaches producing many small tests did not necessarily share that weakness. ACL2 and property-based testing with Proptest also performed well. These findings challenge blanket claims that agents should avoid writing tests, but do not establish which technique is generally best. Much larger evaluations would be needed to determine whether the apparent gains persist across tasks and prompt variations.
Summary generated by AI from the linked article. hn.today is not affiliated with Hacker News or Y Combinator.