ViBench 2.0 Released
The story so far
Replit has released ViBench 2.0, a benchmark for evaluating model capabilities in long-term software construction. The benchmark includes 17 products, 241 feature requests, and 544 test plans, with the top model being GPT-6.1 Sol. All results can be rerun on OpenHands-based harness v2.0.6 with a single command. The CrossSWE benchmark tests the cross-repository consistency of large language models in cross-repository joint software development, including 171 instances covering 165 repositories, 20 technical fields, and 13 programming languages. The evaluation shows that the strongest GPT-6.1 Sol only achieves 58.48% Pass@1, with cross-repository inconsistency being the main failure mode.
AI-written from the reports · Updated 15 days ago
Developments
2 developments
- 3 Nov, 08:00 · 1 reportsCrossSWE: Benchmarking Cross-Repository Joint Development foHugging Face Daily Papers:CrossSWE: Benchmarking Cross-Repository Joint Development for Coding Agents
- 29 Oct, 02:50 · 1 reportsViBench 2.0 ReleasedReplit Blog:ViBench 2.0: A Benchmark for Long-Horizon Software Building
Coverage timeline
Follow the coverage to see every side of the story.
- Hugging Face Daily PapersCrossSWE: Benchmarking Cross-Repository Joint Development for Coding Agents
CrossSWE benchmark tests the cross-repository consistency of large language models in cross-repository collaborative software development, including 171 instances, covering 165 repositories, 20 technical fields, and 13 programming languages. The evaluation shows that the strongest GPT-6.1 Sol only achieves 58.48% Pass@1, with cross-repository inconsistency being the main failure mode.
- Replit BlogViBench 2.0: A Benchmark for Long-Horizon Software Building
ViBench 2.0 Released, used to evaluate the ability of models in long-term software construction, including 17 products, 241 feature requests, and 544 test plans, top model GPT-6.1 Sol.