OpenDS-World benchmark released
The story so far
qwen released the OpenDS-World benchmark to evaluate autonomous deep search agents in real network environments, showing a significant gap between cutting-edge models and human performance. Among 12 cutting-edge models, the best model achieved an accuracy of 39.0%, while the accuracy of 1000 human evaluations was 21.1%. Analysis indicates that research strategies vary among models, and tool coordination and long-term search management are major challenges. OpenDS-World introduces a real-time network benchmark to evaluate autonomous multi-step deep search agents, revealing a significant gap between cutting-edge models and human performance.
AI-written from the reports · Updated 27 days ago
Coverage timeline
Follow the coverage to see every side of the story.
- Hugging Face Daily PapersDeep Search in Real-World Web Environments
OpenDS-World introduces a real-time network benchmark to evaluate autonomous multi-step deep search agents, revealing a significant gap between cutting-edge models and human performance. Among 12 cutting-edge models, the best model achieved an accuracy of 39.0%, while 1000 human evaluations reached an accuracy of 21.1%. Analysis shows that research strategies vary among models, and tool coordination and long-term search management remain major challenges.