The same model with a different harness: CUGA pass rate 81%, leading
The story so far
The IBM team compared three harnesses, CUGA, Hermes, and Deep Agents, using the same open-source weight model GLM-5.3-flash on 90 tasks in AppWorld, with pass rates of 81.1%, 46.7%, and 30.0%, respectively; the number of scenarios where all three variants passed were 20, 8, and 2. The team claims that simply changing the harness increased task completion rates from 30% to 81%. However, this comparison was conducted by the CUGA team themselves, and CUGA used AppWorld configurations not present in other harnesses, with the three settings not being entirely identical. The results are based on a single benchmark of 90 task samples and may not be generalizable to other tasks; these runs did not use strategies, so strategy execution was not measured.
AI-written from the reports · Updated 15 days ago
Coverage timeline
Follow the coverage to see every side of the story.
- Hugging Face BlogFeaturedSame Model, Different Harness: CUGA, Hermes, and Deep Agents — When to Pick CUGA?
The IBM team compared CUGA, Hermes, and Deep Agents using the same open-source weight model GLM-5.3-flash on 90 tasks in AppWorld, with pass rates of 81.1%, 46.7%, and 30.0%, respectively, and 20, 8, and 2 scenarios where all three variants passed.