Skip to content
Live
Trending storyPast story

AI model costs vary by language

1 reports1 outlets20 days agoUpdate

The story so far

AI overview

When processing different languages, AI models can have a token consumption difference of up to 17 times, with English and other mainstream language users incurring lower costs. The AI Language Fairness Research Coalition states that the implicit 'tokenizer tax' results in users of less common languages on the internet paying higher fees, affecting response speed and context capacity. The coalition recommends that AI labs publish evaluation results for each language and improve tokenizer design to narrow the gap.

AI-written from the reports · Updated 20 days ago

Coverage timeline

Follow the coverage to see every side of the story.

30 Oct
  1. ITHome · AI
    AI model usage costs are affected by different languages, with token consumption varying up to 17 times

    When AI models process different languages, token consumption can vary up to 17 times, with English and other mainstream language users incurring lower costs. The implicit 'tokenizer tax' results in users of less common languages paying higher fees, affecting response speed and context capacity. AI labs need to publish evaluation results for each language and improve tokenizer design to narrow the gap.