Evaluating the performance of coding agents as digital employees
The story so far
The SWE-CoWork evaluation found that current coding agents struggle to track requirements in software engineering tasks, although some tests were passed, the proportion of tasks completed was low. This benchmark includes 101 real code repositories, 11 programming languages, and 12 application domains of development tasks, where agents need to complete requirement understanding, implementation, and validation in collaboration with 26 colleagues. Hugging Face Daily Papers published this evaluation, pointing out the difficulties coding agents face in requirement understanding, implementation, validation, and handover.
AI-written from the reports · Updated 18 minutes ago
Coverage timeline
Follow the coverage to see every side of the story.
- Hugging Face Daily PapersSWE-CoWork: Benchmarking Coding Agents as Digital Employees in Organizations
SWE-CoWork evaluation found that current coding agents struggle to track requirements in software engineering tasks, although some tests passed, the proportion of tasks completed was low. This benchmark includes 101 real code repositories, 11 programming languages, and 12 application domains, with agents needing to complete requirement understanding, implementation, and validation in collaboration with 26 colleagues.
Heat over time
Not enough data yet to draw a trend.