How to Benchmark LLM with OpenRouter
Running the same tasks on OpenRouter, GPT-6.1 Sol passed 180/180 tests in 60 synthetic tasks, DeepSeek had the lowest cost passing 157/180 times, but the correlation with the public ranking is low (Spearman correlation coefficient 0.109). Need to check the number of task repetitions, the number of answers, and billing verification.























