An Empirical Study of Harness Design for Coding Agents

This study isolates how three harness components - planning, action space, and context management - shape autonomous coding agent performance by holding the execution loop constant while varying those components. Four models were evaluated on SWE-Bench Verified and Terminal-Bench 2.1 across 176 matched settings that span five context-management strategies, four context-window budgets, and targeted ablations of planning and action-space design. The experimental design measures accuracy, cost, and trajectory behavior to reveal which harness elements drive gains versus overheads.

Key findings: context management grows more valuable as context budgets shrink, primarily by preventing context-overflow failures; the most efficient strategy is rule-based elision staged before LLM summarization, while making elided content recoverable adds complexity without accuracy gains. Planning functions as an accuracy scaffold for weaker models but becomes a cost reducer for stronger models with little accuracy change. Predefined tools benefit models with poor bash proficiency, whereas bash-capable models operate effectively with a bash-only interface and substantially lower cost on command-line-centric tasks. Trajectory-level analysis shows context management lengthens execution without altering behavior, planning shifts termination points of trajectories, and action space adjusts the granularity of code generation. The results offer concrete, model- and budget-aware harness design recommendations and a modular evaluation framework for future components.