hn.today

The Efficiency-Throughput Gap with GitHub Copilot

cacm.acm.org7 points10 comments
Screenshot of The Efficiency-Throughput Gap with GitHub Copilot

Commenters debated a study claiming GitHub Copilot didn’t increase engineering throughput despite some perceived benefits. sublimefire quoted the paper’s timeline and findings - baseline April-June 2024, post-Copilot Sept-Nov 2024, reported reductions in time spent and higher motivation but no statistical improvements in pull requests or lines of code through May 2025 - and argued the work is out of date. aurareturn and njaa pointed out specific timeline/model concerns (suggesting older Sonnet releases were in use) and criticized the authors’ choice of title given when the data were collected. VCFundedGenYer dismissed Copilot’s quality outright as a “sloppy” Microsoft effort, while pu_pe warned that AI evolves too fast for traditional academic timelines.

Opinion split over whether the negative throughput result reflects Copilot’s limits or study design. thevinter and others contended GitHub Copilot is only a limited harness compared with newer tools (thevinter also praised Claude Cowork), and blamed unspecified model versions and lack of retrieval-augmented workflows for poor results. erikgahner questioned sample size and generalizability, and jbjbjbjb reminded readers that coding metrics capture only part of engineers’ work. Overall, some commenters treated the findings as outdated or narrowly scoped, while others accepted the measured gap but urged deeper investigation of modern integrations and broader productivity measures.

Read on cacm.acm.org10 comments on Hacker News

Summary generated by AI from the linked article. hn.today is not affiliated with Hacker News or Y Combinator.

More in AI

The daily digest

Today's best Hacker News stories, summarized and screenshotted, one email a day.