Co-Design Bench released: Encoding agents only fully solve 26.4% of tasks
The story so far
Alibaba and researchers from several Asian universities released the coding agent benchmark Co-Design Bench, which includes 273 tasks and 2,331 acceptance criteria written by experts, based on 26,314 real user coding sessions. Among 10 tested model endpoints, the best-performing GPT-6 Astra meets about 70% of the criteria for each task on average, but only 26.4% of tasks fully meet all criteria; Qwen3.8-27B has an overall strict completion rate of 64.4% and ranks second in the warehouse editing subset. The benchmark evaluates only one fixed request per task and does not include user follow-up questions or mid-session corrections; the report also notes that the AI judge used for scoring may make errors.
AI-written from the reports · Updated 10 days ago
Coverage timeline
Follow the coverage to see every side of the story.
- The DecoderFeaturedAI coding agents still need human cleanup on nearly three out of four tasks in a real-world test
Researchers from Alibaba and several Asian universities used real user change requests to build the coding intelligence agent benchmark Co-Design Bench. Among the ten tested models, the best-performing GPT-6 Astra only meets about 70% of the task requirements, but only 26.4% of the tasks are completely correct.
- Hugging Face Daily PapersCo-Design Bench: Multimodal Collaborative Coding Evaluation for Agentic Models
The Co-Design Bench benchmark includes 273 tasks and 2,331 acceptance criteria written by experts, derived from 26,314 real user coding sessions. On 10 model endpoints, gpt-6-astra averages 69.9% passing the criteria for each task, but only 26.4% of tasks are fully satisfied; Qwen3.8-27B has an overall strict completion rate of 64.4% and ranks second in the repository editing subset.