FrontierHarness Eval v1.1 compares 22 coding-agent harness configurations across five language models and 62 terminal tasks, using 27,280 runs and 83 billion tokens. Its central question is whether the best harness depends on the model. The results say yes: cost rankings shifted substantially between models, with the median harness moving 11 of 22 places. Codex CLI ranked cheapest per task with GLM-5.3 but 16th with Kimi-K3; across all models, OpenCode v2 was cheapest, while ZCode achieved the highest pass rate. The best value choices also varied: Codex CLI with GLM-5.3, Pi with GLM Flash, ZCode with DeepSeek, Oh My Pi with Muse, and Alma with Kimi-K3. These comparisons use default harness settings, and some model-specific winners were close.
Model choice explained most overall cost variation, but harness choice still mattered: for a fixed model, the most expensive harness cost 1.8 to 5 times as much as the cheapest. Higher spending did not reliably buy higher pass rates. Instead, the number of agent steps - and, within models, tool calls - tracked cost; prompt size, tool count, and cache-hit rate did not. Across trials, cached input accounted for 42% of inference cost, output for 33%, and uncached input for 25%. Most strikingly, 64.5% of scored-run spending went to trials that failed or timed out. The findings suggest developers should evaluate harnesses alongside models and focus on efficient agent trajectories, while recognizing that results reflect specific tasks, defaults, and list-price estimates.
Summary generated by AI from the linked article. hn.today is not affiliated with Hacker News or Y Combinator.