Skip to content
Live
Trending storyPast story

ViBench 2.0 Released

2 reports2 outlets16 days agoUpdate

The story so far

AI overview

Replit has released ViBench 2.0, a benchmark for evaluating model capabilities in long-term software construction. The benchmark includes 17 products, 241 feature requests, and 544 test plans, with the top model being GPT-6.1 Sol. All results can be rerun on OpenHands-based harness v2.0.6 with a single command. The CrossSWE benchmark tests the cross-repository consistency of large language models in cross-repository joint software development, including 171 instances covering 165 repositories, 20 technical fields, and 13 programming languages. The evaluation shows that the strongest GPT-6.1 Sol only achieves 58.48% Pass@1, with cross-repository inconsistency being the main failure mode.

AI-written from the reports · Updated 15 days ago

Developments

2 developments

  1. 3 Nov, 08:00 · 1 reports
    CrossSWE: Benchmarking Cross-Repository Joint Development fo
    Hugging Face Daily Papers:CrossSWE: Benchmarking Cross-Repository Joint Development for Coding Agents
  2. 29 Oct, 02:50 · 1 reports
    ViBench 2.0 Released
    Replit Blog:ViBench 2.0: A Benchmark for Long-Horizon Software Building

Coverage timeline

Follow the coverage to see every side of the story.

3 Nov
  1. Hugging Face Daily Papers
    CrossSWE: Benchmarking Cross-Repository Joint Development for Coding Agents

    CrossSWE benchmark tests the cross-repository consistency of large language models in cross-repository collaborative software development, including 171 instances, covering 165 repositories, 20 technical fields, and 13 programming languages. The evaluation shows that the strongest GPT-6.1 Sol only achieves 58.48% Pass@1, with cross-repository inconsistency being the main failure mode.

29 Oct
  1. Replit Blog
    ViBench 2.0: A Benchmark for Long-Horizon Software Building

    ViBench 2.0 Released, used to evaluate the ability of models in long-term software construction, including 17 products, 241 feature requests, and 544 test plans, top model GPT-6.1 Sol.