Random Is Almost Enough? Stable and Low-Cost Benchmark Compression via BestRand
Random Is Almost Enough? Stable and Low-Cost Benchmark Compression via BestRand
BestRand confirms high-probability strong performance of random sampling in benchmark tests through analysis of error distribution, but exists rare large errors leading to instability. The method is validated in over 70 text, multimodal, and agent benchmark tests, reducing high error risk by 50.2% compared to competing methods, decreasing average estimated error by 16.4%, while requiring only 51.3% of the construction cost, and averaging a 21.8% reduction in OOD estimated error.
Abstract
Benchmark compression reduces the often prohibitive time and cost of model evaluation by evaluating models on a small benchmark subset that closely matches full-benchmark results. To quantify estimation error, we derive an accurate analytical approximation to the error distribution when estimating full-benchmark scores from a subset, showing that most subsets yield small errors, while a few produce much larger errors, resulting in a long-tailed distribution. This means random sampling alone can achieve surprisingly strong performance with high probability, but remains unstable due to rare large errors. We further show that this instability is inherent to modern foundation models and persists even as more models are evaluated. Based on these findings, we introduce BestRand, a stable and low-cost benchmark compression method that preserves the efficiency of random sampling while substantially reducing its high-error tail. We validate BestRand on 70+ text, multimodal, and agent benchmarks, and further extend it to VLA evaluation. Compared with competitive methods, BestRand reduces high-error risk by 50.2% and estimation error by 16.4% on average, while requiring only 51.3% of the construction cost. It further reduces OOD estimation error by 21.8% on average across score-range, model-family, and temporal shifts. Together, our theory provides practical guidance for understanding and predicting the accuracy and reliability of benchmark compression, while BestRand enables stable and accurate compression at low cost. The code and data are publicly available onhttps://github.com/George-Ling3/BestRand.
View arXiv page View PDF GitHub 3 Add to collection
Models citing this paper 0
No model linking this paper
Cite arxiv.org/abs/2611.00883 in a model README.md to link it from this page.
Datasets citing this paper 0
No dataset linking this paper
Cite arxiv.org/abs/2611.00883 in a dataset README.md to link it from this page.
Spaces citing this paper 0
No Space linking this paper
Cite arxiv.org/abs/2611.00883 in a Space README.md to link it from this page.
Collections including this paper 0
No Collection including this paper
Add this paper to a collection to link it from this page.
Source: Hugging Face Daily Papers · huggingface.co