Skip to content
Live
ITHome · AI·· 16 days ago

AI model usage costs are affected by different languages, with token consumption varying up to 17 times

Brief overview

When AI models process different languages, token consumption can vary up to 17 times, with English and other mainstream language users incurring lower costs. The implicit 'tokenizer tax' results in users of less common languages paying higher fees, affecting response speed and context capacity. AI labs need to publish evaluation results for each language and improve tokenizer design to narrow the gap.

Source: ITHome · AI · ithome.com