AI model costs vary by language
The story so far
When processing different languages, AI models can have a token consumption difference of up to 17 times, with English and other mainstream language users incurring lower costs. The AI Language Fairness Research Coalition states that the implicit 'tokenizer tax' results in users of less common languages on the internet paying higher fees, affecting response speed and context capacity. The coalition recommends that AI labs publish evaluation results for each language and improve tokenizer design to narrow the gap.
AI-written from the reports · Updated 20 days ago
Coverage timeline
Follow the coverage to see every side of the story.
- ITHome · AIAI model usage costs are affected by different languages, with token consumption varying up to 17 times
When AI models process different languages, token consumption can vary up to 17 times, with English and other mainstream language users incurring lower costs. The implicit 'tokenizer tax' results in users of less common languages paying higher fees, affecting response speed and context capacity. AI labs need to publish evaluation results for each language and improve tokenizer design to narrow the gap.