Skip to content
Live
Trending storyPast story

Research on the encoding proxy framework shows that the cost increase effect remains unchanged

1 reports1 outlets25 days agoUpdate

The story so far

AI overview

Liu and Han's research shows that complex coding agent frameworks did not improve performance but increased costs. The Claude Code, OpenCode, and mini-SWE-agent three frameworks increased the number of tokens consumed per model by two to three times, but the pass rate remained within the measurement uncertainty range. OpenCode showed statistically significant lag in large task sets, mainly due to output limits and shorter runtime limits, while the system prompts and tool descriptions of other frameworks were the main sources of cost differences. The study indicates that the performance gap in previous research might be due to noise from single-run measurements and that all runs were conducted offline.

AI-written from the reports · Updated 25 days ago

Coverage timeline

Follow the coverage to see every side of the story.

24 Oct
  1. The Decoder
    Fancy harnesses make coding agents more expensive but not better, study finds

    Research indicates that the performance gap between different coding agent frameworks is within the range of statistical noise, and the effect of switching frameworks is comparable to rerunning with the same settings. Claude Code, OpenCode, and mini-SWE-agent three frameworks increase the number of tokens consumed per model by two to three times, but the pass rate is still within the range of measurement uncertainty. OpenCode shows statistically significant underperformance in large task sets, mainly due to output limits and shorter runtime limits, while system prompts and tool descriptions in other frameworks are the main sources of cost differences.