hn.today

Memdoor: A team of 5 Qwen3.8-27B models beats Fable 5 on LiveCodeBench Hard

github.com120 points63 comments
Screenshot of Memdoor: A team of 5 Qwen3.8-27B models beats Fable 5 on LiveCodeBench Hard

Memdoor’s team-solve is a reusable workflow for tackling programming problems with multiple AI agents. A planner, two ideators, a solver, a manager and a repair worker operate in fresh contexts, exchanging plans, proposals, code and review instructions through files rather than chat. Commands run sample tests, but the final command - not a model’s judgment - determines whether the solution passes. Users can install the workflow, provide a task and paired input/output samples, and run it with a configured model. The workflow is intended to reproduce the method described by GVS5H, including its prompts and iterative process.

In a five-pass test on the first ten problems in GVS5H’s LiveCodeBench Hard set, the Qwen3.8-27B workflow solved 49 of 50 runs and averaged 9.8 out of 10, compared with 9.6 for the authors’ 27B harness and 8.0 for Fable 5 alone. These results cover only ten problems, not the authors’ 100, and Fable 5 was evaluated from the authors’ results rather than in the same harness. The runs used OpenRouter-hosted models, cost $35.79 in total, and took a median of 40 minutes each; training-data contamination cannot be ruled out. Failures included a manager overturning a correct solution based on a flawed counterexample and later rounds replacing solutions that already passed hidden tests. The write-up identifies safeguards for both problems but says they were not included in the reported scores.

Read on github.com63 comments on Hacker News

Summary generated by AI from the linked article. hn.today is not affiliated with Hacker News or Y Combinator.

More in AI

AI Made Me Homeless

AI Made Me Homeless

Software engineer and consultant Eric Elliott says his mentoring and consulting business collapsed as generative AI reshaped demand, leaving him homeless despite decades of experience and AI expertise. He argues that learning AI alone cannot protect workers from displacement and warns of broader economic fallout. (medium.com)

The daily digest

Today's best Hacker News stories, summarized and screenshotted, one email a day.