Odysseys2: a long-horizon web agent benchmark

• Lawrence Jang (CMU)
• Alexander Yue (Browser Use)
• Magnus Müller (Browser Use)
• Gregor Zunic (Browser Use)
• Andrew Keunwoo Jang (CMU)
• Younghan Park (CMU)
• Mareks Woodside (CMU)
• Daniel Fried (CMU)
• Jing Yu Koh (CMU)
• Ruslan Salakhutdinov (CMU)

Oct 27th, 2026Research

Odysseys2 is a long-horizon web agent benchmark from Carnegie Mellon University and Browser Use: 100 long tasks on the live web, graded against 982 pass/fail checks. The best agent, GPT-6 Astra, fully completes 47 of 100 tasks. It builds on Odysseys, Carnegie Mellon University's original long-horizon benchmark.

The result

The best agent fully completes 47 of 100 tasks. GPT-6 Astra (default effort) has a mean rubric score of 84.6, yet falls short on the other 53.

We ran 15 agents, one 60-minute attempt per task. A task counts as fully completed only when every one of its 8 to 12 checks is met. The mean rubric score is the share of checks met (0 if the attempt is flagged for suspected reward hacking).

Opus 5.5 is second with 18 tasks fully completed and a mean of 71.2. Five of the 15 agents fully complete none. Every agent used BrowserCode, Browser Use's harness that drives the browser by writing code against the Chrome DevTools Protocol. Astra ran at default effort; Opus 5.5, Grok 4.7, GPT-5.6 Luna and Gemini 3.8 Flash at high. Fable 5's content filters blocked 2 tasks, so its mean uses 98.

What a task looks like

These are long, multi-part jobs, not quick lookups. The median instruction is 271 words. Most ask for several deliverables, graded on 8 to 12 checks.

A task reads like a brief to a research assistant: a spreadsheet, a log of pages that failed, a short written comparison, plus things the agent must not do, like signing in. Every task starts from a Google search page. Agents ran in Browser Use's stealth browsers with CAPTCHA solving.

How tasks were built

Tasks are built from the real user request distribution provided by Browser Use. Source requests were de-identified and no request was copied one-to-one. Tasks had to be rerunnable, so anything that changes a site's state was dropped. Four people reviewed every task and rubric, and each went through five rounds of model-assisted alignment and feasibility checks.

Carnegie Mellon University built the original Odysseys benchmark and its rubric method; Browser Use provided the source request distribution.

How runs are graded

One LLM call grades each attempt: GPT-5.6 Luna at xhigh effort, given the task, rubric, action trajectory, final response, output files and selected screenshots. It browses nothing. A check is met only when the delivered content is backed by evidence in the trace; data recited from memory earns nothing. If the judge suspects reward hacking, including fabrication, the attempt scores 0.

8 of Luna's 100 high-effort attempts were flagged, more than Opus 5.5 (1) or Grok 4.7 (2), even with Luna as the judge. Gemini 3.8 Flash was flagged on 75.

Fast is not good

Fast is not the same as good. GPT-5.6 Luna leads for the first half hour, then Grok 4.7 and Opus 5.5 pass it and keep climbing.

By 15 minutes Luna has 37 points, Opus 5.5 18 and Grok 4.7 12. Grok catches up at about 30 minutes, Opus at about 36. Luna's median attempt takes 13 minutes and costs $0.35, against about 25 minutes and $6.22 for Opus 5.5 (model plus judge).

The harness matters

Same model, different harness, different score. At high effort, GPT-5.6 Luna scores 33.0 in the Browser Use Python library and 59.7 in BrowserCode.

We ran Luna in the Browser Use Python library, BrowserCode and Pi. The Python library stays around 30. BrowserCode and Pi land close at high effort (59.7, 59.2) and split at max (63.0, 71.9). Harnesses also hand the judge different records, so these gaps mix execution and evidence.

Same agent, eight runs

One run is not the whole story. Across eight runs of GPT-5.6 Luna, 26 of 100 tasks were fully completed at least once, and none in all eight.

Luna's mean across eight runs is 56.9 (standard deviation 2.9). Each task's best of eight averages 83.1; its worst, 23.1. The runs differ in dates, concurrency and logging, so the spread includes those conditions.

Grading changes scores

How you grade decides who looks good. On the same attempts, Gemini 3.7 Flash scores 94.9 under the original Odysseys judge and 5.8 under the Odysseys2 judge.

The original Odysseys judge checks one item at a time from actions and screenshots, without the delivered files. On the same attempts from 14 agents, its means run 77 to 96; the Odysseys2 judge's run 2 to 85. Astra is first under both. This measures disagreement between judges, not which is more accurate.

Why we care

Our view, not a measured result: the browser work people automate is usually a brief, not a lookup. Collect data across many sites, follow constraints, deliver a spreadsheet and a summary. Astra meets most checks on average and still misses something on 53 of 100 tasks, and for the person relying on it, one missed detail can mean redoing the job.

We will run new models on Odysseys2 next to Browser Use Bench v2.1. More on long-horizon RL environments and the benchmark behind the benchmark.

Limitations

• Judge not validated against a reliable human reference; the paid human-review pilot was insufficient.
• Flags are model judgments, not proof of fabrication or intent. Zeroing the whole attempt is a policy choice.
• Not fully reproducible. Live sites change, and the runner, judge implementation and trajectories are not released.
• Clipped evidence. Long tool results and file previews can be clipped; the judge does not see image pixels.
• Settings and costs differ. Astra ran at default effort and its flag status is unavailable. Costs for the four high-effort agents include the judge; others are model only.
• Intervals are as reported in the release, not recomputed. The eight-run spread is one standard deviation, not a confidence interval.

Get the data

• Benchmark page and interactive results: odysseysbench.com/v2
• 100 tasks as JSON (instruction, start URL, rubric)
• Judge prompt and EVALUATION.md
• Release files (tasks, rubrics, scores, judge prompt): github.com/ljang0/odysseys2-benchmark (MIT license)
• Original Odysseys paper: arXiv 2604.24964
• Related: web agent benchmarks, the bitter lesson of browser agents

To cite Odysseys2:

@misc{jang2026odysseys2,
  title        = {Odysseys2: A Web Agent Benchmark Based on Real User Requests},
  author       = {Jang, Lawrence and Yue, Alexander and M{\"u}ller, Magnus and
                  Zunic, Gregor and Jang, Andrew Keunwoo and Park, Younghan and
                  Woodside, Mareks and Fried, Daniel and Koh, Jing Yu and Salakhutdinov, Ruslan},
  year         = {2026},
  howpublished = {\url{https://odysseysbench.com/v2/}}
}

Thanks to our co-authors at Carnegie Mellon University: Lawrence Jang, Andrew Keunwoo Jang, Younghan Park, Mareks Woodside, Daniel Fried, Jing Yu Koh and Ruslan Salakhutdinov.