Itch.io adding Generative AI disclosure filtering to few more places on the site
Itch.io is expanding its filters for games and other content disclosed as using generative AI. The filters will appear in more places across the site. (bsky.app)
When Modernization Breaks Working Code: A Study of LLMs on Legacy PHP
arxiv.org21 points14 comments
The study tests whether large language models can safely modernize legacy PHP when given only a single file, without project context or feedback from running the code. Five models were asked to update 100 files from WordPress 4.0 for PHP 8.3 while preserving behavior. Researchers assessed the results using 630 hidden behavioral checks and 511 modernization findings detected by Rector, comparing model outputs with both unchanged files and Rector’s rule-based transformations.
The untouched code performed better than the models on behavioral preservation: 83 files passed every check before modification, while the best model left only 59 passing. Each model broke 32-51 previously working files and repaired no more than 10; Rector broke 7 and repaired 6. Only 37 modernization findings - 7 percent of the total - required changes for PHP 8.3. Even among files that needed no change, models broke 36-60 percent, compared with 8 percent for Rector. Many failures made files impossible to load, often through PHP-rejected type declarations. The model with the strongest modernization score was not more behaviorally compatible than its competitors. The findings show that modernization quality and preservation of existing behavior are distinct measures, and evaluations should compare changes against doing nothing.
Summary generated by AI from the linked article. hn.today is not affiliated with Hacker News or Y Combinator.
More in AI
Sam Altman's Case Against Your RAM Is Efficient. That's the Problem
Sam Altman argues that people should use less RAM so memory can be redirected to AI data centers. Critics say OpenAI’s efforts to secure a large share of global RAM supplies could also drive up prices and question whether efficiency justifies restricting personal use. (techdirt.com)
AI agents hallucinate prices: LLM grounding with live data
LLMs often give outdated or incorrect product prices, even when they use web search. The tutorial shows how to build a Python agent that grounds answers in live data and lets code insert verified prices. (blog.apify.com)
Revolut moves to self-hosted GLM
Revolut moved about 2,000 engineers to self-hosted GLM coding models, running them on 56 Nvidia B200 GPUs at roughly $0.30 per million tokens. AI operations lead David Abitbol says engineering lead time improved 70% as lower costs let teams use AI more broadly, while Claude Code and Codex became exceptions. (twitter.com)
Could a modern chess engine on a Pentium have defeated Kasparov in 1996?
A 1995 Pentium Pro running a modern Stockfish port is estimated to play at 3300 Elo, far above Kasparov’s peak rating. The experiment shows how advances in chess algorithms could have beaten Kasparov without Deep Blue’s custom supercomputer hardware. (nicholas.carlini.com)
Open models can do 90 percent of agentic work at a twentieth of the cost
Gartner says open-weight models can handle about 90% of routine enterprise agent tasks at a fraction of the cost of closed models, while frontier models remain useful for complex reasoning. It expects companies to rely more on specialized open models, making the software harnesses that route tasks between models a key source of value. (theregister.com)