Artificial Analysis v5 released, evaluation benchmarks still face challenges
The story so far
Artificial Analysis has released Intelligence Index v5, adjusting the evaluation weight distribution, with knowledge work, coding, science, and general abilities each accounting for 25%. The latest rankings show GPT 6.1 Terra outperforming Claude Fable 5.1. The 60% weight of this evaluation index is based on private test sets or hidden answers, aiming to reduce the 'inflation' of tests caused by publicly available question banks being contaminated by model training data. The evaluation still includes nine tests, with the coding portion weighted towards Cognition's FrontierCode 2.0 (15%) and Terminal-Bench 4.0 (10%). DeepSeek V4.1 Flash 1017 scored 28% on FrontierCode 2.0, leading open-source models, but its comprehensive ranking still lags behind Claude Opus 5.5 by 26 points. Nathan Lambert stated that improving score design is expensive, and creating a true cutting-edge AI evaluation benchmark is even more difficult.
AI-written from the reports · Updated 20 days ago
Developments
2 developments
- 29 Oct, 01:06 · 1 reportsArtificial Analysis Intelligence Index v5 benchmark launchedITHome · AI:Artificial Analysis Intelligence Index v5 benchmark launched, DeepSeek V4.1 Flash 1017 achieves the highest score for open-source models on FrontierCode 2.0.
- 28 Oct, 20:58 · 1 reportsArtificial Analysis v5 and the benchmarker's dilemmaInterconnects:Artificial Analysis v5 and the benchmarker's dilemma
Coverage timeline
Follow the coverage to see every side of the story.
- ITHome · AIArtificial Analysis Intelligence Index v5 benchmark launched, DeepSeek V4.1 Flash 1017 achieves the highest score for open-source models on FrontierCode 2.0.
Artificial Analysis Intelligence Index v5 released, 60% review weight is based on a private test set for the first time, DeepSeek V4.1 Flash 1017 leads open-source models with 28% score on FrontierCode 2.0, but still lags behind Claude Opus 5.5 by 26 points.
- InterconnectsArtificial Analysis v5 and the benchmarker's dilemma
Artificial Analysis released Intelligence Index v5, an independent evaluation metric with weight distribution of knowledge work (25%), coding (25%), science (25%), and general (25%). The latest model rankings show GPT 6.1 Terra outperforming Claude Fable 5.1.