Skip to content
Live
Trending storyPast story

Longtail-SWE released

1 reports1 outlets9 days agoUpdate

The story so far

AI overview

Hugging Face has released the Longtail-SWE benchmark, which includes 202 problem-solving tasks covering 13 underrepresented programming languages, aimed at addressing the scarcity of engineering problems in existing benchmarks. The benchmark is constructed through a solver-judge-repairer pipeline and evaluates model performance under 9 configurations. All benchmark instances, execution environments, tools, and agent trajectories are open-sourced.

AI-written from the reports · Updated 8 days ago

Coverage timeline

Follow the coverage to see every side of the story.

10 Nov
  1. Hugging Face Daily Papers
    Longtail-SWE: High-Quality SWE Tasks for Underrepresented Languages

    Longtail-SWE has released 202 problem-solving tasks covering 13 underrepresented programming languages, for engineering problems that are scarce in existing benchmarks. Built through a solver-judge-repairer pipeline, the model was evaluated under 9 configurations, including state-of-the-art models, analyzing task-solving performance, consistency of repeated attempts, and resource usage. All benchmark instances, execution environments, tools, and agent trajectories have been open-sourced.

Heat over time

Not enough data yet to draw a trend.