ITHome · AI·· 16 days ago
AI model usage costs are affected by different languages, with token consumption varying up to 17 times
Brief overview
When AI models process different languages, token consumption can vary up to 17 times, with English and other mainstream language users incurring lower costs. The implicit 'tokenizer tax' results in users of less common languages paying higher fees, affecting response speed and context capacity. AI labs need to publish evaluation results for each language and improve tokenizer design to narrow the gap.
Source: ITHome · AI · ithome.com