Skip to content
Live
Trending storyPast story

CWE-bench v2 Released

1 reports1 outlets16 days agoUpdate

The story so far

AI overview

Collinear AI has released CWE-bench v2, enhancing coverage, quality, and transparency, and opening 100 tasks for public model evaluation, covering 91 CWEs. The scoring process is programmable and reproducible, and anyone can freely reproduce existing results. All programmable checks are assisted by an LLM judge panel to ensure difficulty and fairness. Additionally, a set of unseen tasks is maintained to continuously track model progress on these tasks. Results are tracked in two places on the Hugging Face Hub, with the CWE-bench dataset page leaderboard covering open-weight models and automatically updating.

AI-written from the reports · Updated 15 days ago

Coverage timeline

Follow the coverage to see every side of the story.

3 Nov
  1. Hugging Face Blog
    CWE-bench v2: More coverage, quality, and transparency

    CWE-bench v2 opens 100 tasks for public model evaluation, covering 91 types of CWE, with a scoring system that is programmable and reproducible. Gemini 4 Argon performed the best among 11 models, leading with 67.8% pass@1 and 81.0% pass@4, but still failed 19% of the tasks. DeepSeek V4.1 Flash forms the cost frontier with a cost of $0.04 per inference.