AI models cost more when you don't use English

AI text systems divide input into tokens, and a study by the Coalition for Language Equity in AI Research finds that this process disproportionately burdens many non-English languages. Among the 20 most-used internet languages, some require 1.5 to five times as many tokens as others; across 200 languages, the gap reaches 17 times in one model. More tokens can mean higher usage costs, slower responses, and less room for documents or conversation history. That loss of context can also weaken services such as tutoring, where retaining a learner’s previous questions and mistakes matters. The researchers call this imbalance a “tokenizer tax,” warning it can disadvantage businesses and users in places where computing resources and internet access are already constrained.

Language rarity alone does not explain the differences: writing system matters too. Latin-script languages can benefit from tokenizers’ familiarity with related languages, while Burmese, Tibetan, Khmer, Lao, Tifinagh, and Ol Chiki are consistently poorly served. Better digital training data, including speech and text gathered from native speakers, can improve language coverage, but will not by itself correct tokenization. Developers must design tokenizers to handle more scripts efficiently. Although falling token prices may soften the immediate financial burden, unequal token use still affects speed, context, and quality. Researchers and industry leaders argue that language efficiency should be treated as a product requirement, with AI companies publishing comparable language-by-language performance and cost measures and setting targets to improve their weakest results.