CWE-bench v2 Released
The story so far
Collinear AI has released CWE-bench v2, enhancing coverage, quality, and transparency, and opening 100 tasks for public model evaluation, covering 91 CWEs. The scoring process is programmable and reproducible, and anyone can freely reproduce existing results. All programmable checks are assisted by an LLM judge panel to ensure difficulty and fairness. Additionally, a set of unseen tasks is maintained to continuously track model progress on these tasks. Results are tracked in two places on the Hugging Face Hub, with the CWE-bench dataset page leaderboard covering open-weight models and automatically updating.
AI-written from the reports · Updated 15 days ago
Coverage timeline
Follow the coverage to see every side of the story.
- Hugging Face BlogCWE-bench v2: More coverage, quality, and transparency
CWE-bench v2 opens 100 tasks for public model evaluation, covering 91 types of CWE, with a scoring system that is programmable and reproducible. Gemini 4 Argon performed the best among 11 models, leading with 67.8% pass@1 and 81.0% pass@4, but still failed 19% of the tasks. DeepSeek V4.1 Flash forms the cost frontier with a cost of $0.04 per inference.