OpenRouter released a benchmark test guide for LLMs.
The story so far
OpenRouter has released a guide on how to benchmark large language models. Running the same tasks on OpenRouter, GPT-6.1 Sol passed 180/180 tests in 60 synthetic tasks, DeepSeek had the lowest cost by passing 157/180 times, but the correlation with the public ranking is low (Spearman correlation coefficient 0.109). Need to check the number of task repetitions, the number of answers, and billing verification.
AI-written from the reports · Updated 3 days ago
Coverage timeline
Follow the coverage to see every side of the story.
- OpenRouter BlogHow to Benchmark LLM with OpenRouter
Running the same tasks on OpenRouter, GPT-6.1 Sol passed 180/180 tests in 60 synthetic tasks, DeepSeek had the lowest cost passing 157/180 times, but the correlation with the public ranking is low (Spearman correlation coefficient 0.109). Need to check the number of task repetitions, the number of answers, and billing verification.
Heat over time
- Now
- 3
- Peak
- 1012 Nov, 09:00
- 24h change
- -49%
Hover or tap the chart to see each hour; use the arrow keys on a keyboard.