Hacker News · 300+· antiuniverse·· 7 days agoFeatured71
MaxxBench – "Nice DeepSWE score. But is it true?"
MaxxBench – "Nice DeepSWE score. But is it true?"
Brief overview
The MaxxBench team made minor changes to 89 DeepSWE tasks (median reference patch added only 17 lines, 3.5%), re-evaluating 15 model configurations, finding the difference between the highest and lowest scores expanded from 20.9 percentage points in DeepSWE to 48.3 percentage points.
Why it matters
By rewriting the instructions and starting code for 89 DeepSWE tasks, it demonstrates the huge gap in model scores between familiar benchmarks and tasks after modification.
Source: Hacker News · 300+ · maxxbench.unlap.ai