A comparison of 3,000 parallel sentences across English, Swedish, Norwegian, Danish, and Finnish found that all twelve tested frontier AI models used more tokens for the Nordic languages than for English. Across models, Swedish cost 29-68% more on news and 32-68% more on encyclopedic text; Norwegian and Danish premiums were similarly large, while Finnish cost 54-100% more. Because models bill by token and generate one token at a time, these differences raise both usage costs and response times. The study used two public translation datasets and compared token counts for equivalent content, reporting premiums relative to English.
The extra cost comes mainly from tokenizers, which split text into pieces based on vocabularies shaped by training data: common English words are often single tokens, while less-represented Nordic words get split into several. Swedish and Norwegian translations were virtually the same length as English, so wordiness does not explain their premium. A tokenizer designed for Nordic text, used by the open Viking model, reduced the premium to about 5% for Swedish, Norwegian, and Finnish and 7% for Danish. Newer, larger vocabularies have narrowed the gap, but many leading models still rely on vocabularies that underrepresent Nordic text. The findings support investment in local AI labs and language-specific models, which could improve cost efficiency without sacrificing capability.
Summary generated by AI from the linked article. hn.today is not affiliated with Hacker News or Y Combinator.
Anthropic’s IPO filing says future models may soon meet its definition of recursive self-improvement, while none of its current models have reached that point. Its published data shows Claude leads some research tasks but remains fully supervised, with humans still choosing research goals and evaluating results. (epyhia.com)
Astra improved Qwen’s math performance by running the network for 50 passes and editing its weights, not by simply inspecting raw weights. Similar experiments used different methods, and an attempt to improve Gemma 3 failed. (honnibal.dev)
Itch.io is expanding its filters for games and other content disclosed as using generative AI. The filters will appear in more places across the site. (bsky.app)
Sam Altman argues that people should use less RAM so memory can be redirected to AI data centers. Critics say OpenAI’s efforts to secure a large share of global RAM supplies could also drive up prices and question whether efficiency justifies restricting personal use. (techdirt.com)
Harness engineering improves coding-agent reliability by building better systems for documentation, context loading, validation, and feedback instead of relying on better prompts. The excerpt also outlines an AI testing strategy for agent systems, retrieval-augmented systems, and external integrations. (deepsense.ai)
LLMs often give outdated or incorrect product prices, even when they use web search. The tutorial shows how to build a Python agent that grounds answers in live data and lets code insert verified prices. (blog.apify.com)
Today's best Hacker News stories, summarized and screenshotted, one email a day.