Longtail-SWE released
The story so far
Hugging Face has released the Longtail-SWE benchmark, which includes 202 problem-solving tasks covering 13 underrepresented programming languages, aimed at addressing the scarcity of engineering problems in existing benchmarks. The benchmark is constructed through a solver-judge-repairer pipeline and evaluates model performance under 9 configurations. All benchmark instances, execution environments, tools, and agent trajectories are open-sourced.
AI-written from the reports · Updated 8 days ago
Coverage timeline
Follow the coverage to see every side of the story.
- Hugging Face Daily PapersLongtail-SWE: High-Quality SWE Tasks for Underrepresented Languages
Longtail-SWE has released 202 problem-solving tasks covering 13 underrepresented programming languages, for engineering problems that are scarce in existing benchmarks. Built through a solver-judge-repairer pipeline, the model was evaluated under 9 configurations, including state-of-the-art models, analyzing task-solving performance, consistency of repeated attempts, and resource usage. All benchmark instances, execution environments, tools, and agent trajectories have been open-sourced.
Heat over time
Not enough data yet to draw a trend.