The work studies how an optimizer can keep improving a task-solving system’s harness - the supporting setup around its policy model - when the benchmark or model changes. In conventional harness optimization, the optimizer can inspect the current harness but may not know why earlier changes were made or how well they worked. The proposed approach, Self-Improvement Memory (SIM), gives the optimizer a persistent record organized around recurring harness problems. For each edit, it stores the rationale, expected and observed results, and the evaluation context; later optimization attempts add to that history. The memory supports the optimizer, rather than becoming part of the task-solving policy.
Experiments across four domains compare SIM with continuation methods that rely only on the existing harness or on root-cause information. SIM produces higher final scores in cross-dataset and pairwise cross-model comparisons, suggesting that recorded experience can help identify which past solutions to reuse or revise as conditions shift. In coding, ten independent starting runs are continued in paired comparisons: SIM improves final scores by an average of 2.36 points over harness-only continuation, a statistically significant difference (p=0.003). The central claim is that retaining the reasoning and outcomes behind previous edits makes recursive improvement more effective across related tasks and models, not just within a fixed benchmark setup.
Summary generated by AI from the linked article. hn.today is not affiliated with Hacker News or Y Combinator.