Skip to content
Live
Trending storyPast story

The same model with a different harness: CUGA pass rate 81%, leading

1 reports1 outlets15 days agoUpdate

The story so far

AI overview

The IBM team compared three harnesses, CUGA, Hermes, and Deep Agents, using the same open-source weight model GLM-5.3-flash on 90 tasks in AppWorld, with pass rates of 81.1%, 46.7%, and 30.0%, respectively; the number of scenarios where all three variants passed were 20, 8, and 2. The team claims that simply changing the harness increased task completion rates from 30% to 81%. However, this comparison was conducted by the CUGA team themselves, and CUGA used AppWorld configurations not present in other harnesses, with the three settings not being entirely identical. The results are based on a single benchmark of 90 task samples and may not be generalizable to other tasks; these runs did not use strategies, so strategy execution was not measured.

AI-written from the reports · Updated 15 days ago

Coverage timeline

Follow the coverage to see every side of the story.

3 Nov
  1. Hugging Face BlogFeatured
    Same Model, Different Harness: CUGA, Hermes, and Deep Agents — When to Pick CUGA?

    The IBM team compared CUGA, Hermes, and Deep Agents using the same open-source weight model GLM-5.3-flash on 90 tasks in AppWorld, with pass rates of 81.1%, 46.7%, and 30.0%, respectively, and 20, 8, and 2 scenarios where all three variants passed.