Skip to content
Live
Trending storyPast story

Artificial Analysis v5 released, evaluation benchmarks still face challenges

2 reports2 outlets21 days agoUpdate

The story so far

AI overview

Artificial Analysis has released Intelligence Index v5, adjusting the evaluation weight distribution, with knowledge work, coding, science, and general abilities each accounting for 25%. The latest rankings show GPT 6.1 Terra outperforming Claude Fable 5.1. The 60% weight of this evaluation index is based on private test sets or hidden answers, aiming to reduce the 'inflation' of tests caused by publicly available question banks being contaminated by model training data. The evaluation still includes nine tests, with the coding portion weighted towards Cognition's FrontierCode 2.0 (15%) and Terminal-Bench 4.0 (10%). DeepSeek V4.1 Flash 1017 scored 28% on FrontierCode 2.0, leading open-source models, but its comprehensive ranking still lags behind Claude Opus 5.5 by 26 points. Nathan Lambert stated that improving score design is expensive, and creating a true cutting-edge AI evaluation benchmark is even more difficult.

AI-written from the reports · Updated 20 days ago

Developments

2 developments

  1. 29 Oct, 01:06 · 1 reports
    Artificial Analysis Intelligence Index v5 benchmark launched
    ITHome · AI:Artificial Analysis Intelligence Index v5 benchmark launched, DeepSeek V4.1 Flash 1017 achieves the highest score for open-source models on FrontierCode 2.0.
  2. 28 Oct, 20:58 · 1 reports
    Artificial Analysis v5 and the benchmarker's dilemma
    Interconnects:Artificial Analysis v5 and the benchmarker's dilemma

Coverage timeline

Follow the coverage to see every side of the story.

29 Oct
  1. ITHome · AI
    Artificial Analysis Intelligence Index v5 benchmark launched, DeepSeek V4.1 Flash 1017 achieves the highest score for open-source models on FrontierCode 2.0.

    Artificial Analysis Intelligence Index v5 released, 60% review weight is based on a private test set for the first time, DeepSeek V4.1 Flash 1017 leads open-source models with 28% score on FrontierCode 2.0, but still lags behind Claude Opus 5.5 by 26 points.

28 Oct
  1. Interconnects
    Artificial Analysis v5 and the benchmarker's dilemma

    Artificial Analysis released Intelligence Index v5, an independent evaluation metric with weight distribution of knowledge work (25%), coding (25%), science (25%), and general (25%). The latest model rankings show GPT 6.1 Terra outperforming Claude Fable 5.1.

Related stories