Longtail-SWE: High-Quality SWE Tasks for Underrepresented Languages
Longtail-SWE: High-Quality SWE Tasks for Underrepresented Languages
Longtail-SWE has released 202 problem-solving tasks covering 13 underrepresented programming languages, for engineering problems that are scarce in existing benchmarks. Built through a solver-judge-repairer pipeline, the model was evaluated under 9 configurations, including state-of-the-art models, analyzing task-solving performance, consistency of repeated attempts, and resource usage. All benchmark instances, execution environments, tools, and agent trajectories have been open-sourced.
Published on Nov 6
Authors:
,
Abstract
Challenging repository-level benchmarks remain limited for many programming language ecosystems, where filtering scarce candidates with ambiguous descriptions or defective tests can discard valuable engineering problems. We introduce Longtail-SWE, a benchmark of 202 issue-resolution tasks across 13 languages with limited coverage in existing repository-level benchmarks: Clojure, Dart, Elixir, Erlang, F#, Haskell, Julia, Kotlin, Lua, OCaml, R, Scala, and Swift. Its construction uses a solver--judge--repairer pipeline: a solver attempts each candidate task, and a judge reviews the task together with the solver's trajectory and test results to guide revisions to tests and task descriptions. These revisions aim to correct tests that reject valid alternative implementations or accept incorrect solutions while retaining challenging tasks and preserving their intended requirements. We evaluate nine model configurations, including frontier models, and analyze task-solving performance, consistency across repeated attempts, and resource usage. We release all benchmark instances, executable environments, complete construction and evaluation tooling, and agent trajectories, together with an additional set of easier tasks to support evaluation of smaller models.
View arXiv page View PDF Project page GitHub 2 Add to collection
Get this paper in your agent:
hf papers read 2611.10304
Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash
Models citing this paper 0
No model linking this paper
Cite arxiv.org/abs/2611.10304 in a model README.md to link it from this page.
Datasets citing this paper 2
nebius/Longtail-SWE
Viewer • Updated about 17 hours ago • 471 • 6 • 2
nebius/Longtail-SWE-trajectories
Viewer • Updated about 17 hours ago • 5.45k • 3 • 1
Spaces citing this paper 0
No Space linking this paper
Cite arxiv.org/abs/2611.10304 in a Space README.md to link it from this page.
Collections including this paper 1
Source: Hugging Face Daily Papers · huggingface.co