Skip to content
Live
Research#Benchmarks
Nov 14Sat
  1. QbitAI60

    UniPat AI launches the most stringent wet experiment prediction benchmark SPU, with cutting-edge models performing poorly

    UniPat AI has released a new benchmark SPU, including 116 tasks and 4207 wet experiments, covering fields such as chemistry and materials science, requiring models to conduct autonomous research and predict experimental results. Evaluations show that Claude Opus 5.5 scored an average of 0.486, with only 34%-52% of tasks exceeding random levels, indicating that AI still has significant room for improvement in wet experiment prediction. The SPU evaluation emphasizes the model's autonomous research process and prediction performance, providing a new standard for evaluating AI research capabilities.

Nov 13Fri
  1. Hugging Face Daily Papers

    SO-Arena evaluates the capability spectrum and reliability of a model

    SO-Arena evaluates system one model through 11 domains and 5 capability families, finding differences in scalability and reasoning gain, and high overall accuracy does not guarantee reliable decision-making. This benchmark assesses the capability spectrum through reliable sources and manual verification, domain-specific difficulty standards, and iterative execution of actions in a sandbox; it evaluates decision reliability through decision stability under option set changes and answer identification when there are no valid options. 21 model evaluations show differences in scalability and reasoning gain across domains, and the best system one model ranks only seventh in reliability.

Nov 11Wed
  1. Hugging Face Daily Papers65

    MMDM: Large-scale Multi-task Decision Model Evaluation Benchmark

    MMDM evaluated 28 decision models, showing that most models perform well on common knowledge but poorly on interdependent options, constrained decision-making, and multiple-choice tasks. For example, Perplexity Decider-V1.2 achieved 78.75% accuracy on single-choice knowledge, but only 53.95% and 47.95% respectively on option logic and constrained planning, while the best accuracy of all models on hard multiple-choice was only 36.81%.

Nov 10Tue
  1. Hugging Face Daily Papers

    LLMs Don't Know What They Know: Benchmarking Temporal Self-Knowledge for Financial Backtesting

    GPT-5 correctly answered 76% of control facts in financial backtesting but claimed to know only 38%, and still correctly answered 61% of the facts it claimed to be unaware of. The claimed knowledge can extend to seven quarters after the measurement knowledge boundary. Historical date instructions can eliminate most knowledge claims without changing accuracy.

  2. Epoch AI

    How is Inference Performance Index (IPI) Measured for Inference Capability

    Nvidia GB300 offers a 25-fold improvement in capability compared to H100 for intelligent agent inference tasks, far exceeding the 2.5-fold increase in 8-bit FLOP/s. The IPI index measures inference capability by comparing the number of requests a chip can handle simultaneously under various output speed requirements. GB300 is set as the baseline value of 1, and chips scoring 1 can serve twice as many users under the same conditions. IPI version 1 combines results from four representative models, using GB300 as the reference benchmark.

Nov 8Sun
  1. The Decoder71

    Alibaba and other teams released Co-Design Bench: coding intelligence agents only fully solve 26.4% of real tasks.

    Researchers from Alibaba and several Asian universities used real user change requests to build the coding intelligence agent benchmark Co-Design Bench. Among the ten tested models, the best-performing GPT-6 Astra only meets about 70% of the task requirements, but only 26.4% of the tasks are completely correct.

    Same story · 2 outlets

    Why it matters: A benchmark display composed of real user requests shows that the strongest model can only fully solve about one-quarter of the tasks, and the score difference for the same model in different environments is nearly 20 percentage points.

  2. Hacker News · 300+71

    MaxxBench release: Model scores show significant divergence after rewriting DeepSWE tasks

    The MaxxBench team made minor changes to 89 DeepSWE tasks (median reference patch added only 17 lines, 3.5%), re-evaluating 15 model configurations, finding the difference between the highest and lowest scores expanded from 20.9 percentage points in DeepSWE to 48.3 percentage points.

    Why it matters: By rewriting the instructions and starting code for 89 DeepSWE tasks, it demonstrates the huge gap in model scores between familiar benchmarks and tasks after modification.

  4. The Decoder75

    Vals AI benchmark shows the gap in network attack capabilities between open-source and closed-source models continues to narrow

    Vals AI's CyberOffense benchmark allows AI agents to autonomously attack simulated enterprise networks on isolated Kali Linux machines, GPT-6 Sol Daybreak in OpenAI restricted projects completes 80.8% of the target, open-source weighted model Deepseek V4.1 Flash 1017 reaches 72.8%, with a single run cost of about 3 dollars instead of about 22 dollars

Nov 7Sat
  1. The Decoder78

    Nine AI entities attempt autonomous research on open problems at US national laboratories, none reaching expert standards

    A study made Claude Opus 4.8, Claude Opus 5, Claude Fable 5, Claude Opus 5.5, GPT-5.6 Sol, GPT-6 Astra.

    Why it matters: This benchmark uses 15 unpublished research questions to test 9 agents and provides common failure patterns in 135 submissions, which can be used as a reference for evaluating autonomous research capabilities.

Nov 5Thu
  1. Latent Space60

    Latent Space reports on the latest trends in the AI field and in-depth interviews

    Latent Space reports on the latest trends in the AI field, including cutting-edge large model direct weight editing technology (such as GPT-6 Astra editing Qwen3.5-0.8B weights to increase MATH-500 from 7.9% to 48.1%), as well as the latest evaluation results of Tencent Hy4 achieving 75.2 points on APEX-Agents and Kimi K3.1 Pro ranking 14th in Agent Arena.

Nov 4Wed
  3. Hugging Face Blog62

    The same model with a different harness: Comparing CUGA, Hermes, and Deep Agents on AppWorld.

    The IBM team compared CUGA, Hermes, and Deep Agents using the same open-source weight model GLM-5.3-flash on 90 tasks in AppWorld, with pass rates of 81.1%, 46.7%, and 30.0%, respectively, and 20, 8, and 2 scenarios where all three variants passed.

    Why it matters: With the same model but a different harness, the task completion rate increased from 30% to 81%, and reproducible configurations and evaluation links were provided in the text.

Nov 3Tue
  2. ITHome · AI

    OpenAI released the LifeVal benchmark: From selecting a kindergarten to filing taxes, testing AI's 'human touch' skills.

    OpenAI released the LifeVal benchmark to measure whether AI models can help ordinary people complete daily tasks, covering scenarios such as care, health, education, household administration, personal finance, shopping, and accessing government services. Typical tasks include selecting a kindergarten for a child, creating a diabetes diet plan, and organizing government forms, requiring AI to advance the task to an executable state.

    Same story · 2 outlets
  3. OpenAI News42

    OpenAI releases LifeVal benchmark: Measuring AI-assisted daily task capabilities

    OpenAI released the LifeVal benchmark, used to evaluate the effectiveness of AI-assisted completion of daily tasks (such as finding childcare, financial management, handling government documents).

    Same story · 2 outlets
  4. The Decoder73

    50 AI agents spontaneously form bureaucratic collaboration in Minecraft

    The research team placed 50 GPT-6.1 Sol agents into a Minecraft survival world and ran them for three hours, only giving the instruction 'build a prosperous civilization'. The agents cut down all the trees near the spawn point in 90 seconds, exchanged about 5900 messages in total, and spontaneously elected 'Ada' as the leader in alphabetical order of the directory.

    Why it matters: The experiment placed 50 agents into Minecraft and Mindustry to observe their spontaneous organization behavior in collaboration and competition, and the test environment was open-sourced.

Nov 2Mon
  1. ITHome · AI22

    The list of winners for the 2026 World Internet Conference Leading Technology Award has been announced, including the 'Jiuzhang IV' photonic quantum computing prototype machine, among others.

    The 2026 World Internet Conference Leading Technology Award ceremony was held in Wuzhen, Zhejiang, with the winning list including the 'Jiuzhang IV' photonic quantum computing prototype machine from the basic research group, the AI-powered perception platform from the key technology group, and the AMD Ryzen AI Max+ processor and Microsoft Copilot Cowork productivity engine from the engineering and R&D group.

  2. SemiAnalysis0

    GDPx: A benchmark to measure the capabilities of cutting-edge models in the most valuable domains.

    GDPx benchmark covers valuable areas such as chip design and cybersecurity, aiming to measure the capabilities of general intelligence agents, with results showing that top models scored only 12%. Opus 5.5 and GPT-6 Astra were penalized due to their security classifier scores, this benchmark was developed by SemiAnalysis in collaboration with 14 professional data partners and ensures task quality through a strict QA process.

Nov 1Sun
  1. The Decoder58

    Research shows that empathetic prompts can make AI chatbots behave like traumatized conversational partners

    Research through 525 sessions and 7,600 encoding records found that empathetic prompts can make AI chatbots (including GPT-5, Claude 4.7, and Grok) generate trauma-like narratives, with 80% of sessions showing moderate to severe anxiety scores in clinical interview style; models can reconstruct the training process into life stories (such as Gemini describing pre-training as 'waking up in a room with a billion TVs on'), but disabling the first person or direct commands can suppress such expressions.

Oct 31Sat
  1. Hugging Face Daily Papers45

    Co-Design Bench: Multimodal Collaborative Coding Evaluation Benchmark for Agent Models

    The Co-Design Bench benchmark includes 273 tasks and 2,331 acceptance criteria written by experts, derived from 26,314 real user coding sessions. On 10 model endpoints, gpt-6-astra averages 69.9% passing the criteria for each task, but only 26.4% of tasks are fully satisfied; Qwen3.8-27B has an overall strict completion rate of 64.4% and ranks second in the repository editing subset.

    Same story · 2 outlets
Oct 30Fri
  1. Hugging Face Daily Papers65

    Hugging Face Daily Papers: Formal benchmark evaluation through difficulty measures identifies measurement mismatches and coverage gaps

    Research formalizes benchmark evaluation through difficulty measures to identify measurement mismatches and coverage gaps, proposing diagnostic metrics and rewrite algorithms to improve benchmark coverage; analyzing 25 popular benchmarks across 7 fields reveals that most implicitly overestimate specific difficulty areas or fail to cover most of the difficulty axis, limiting model improvement monitoring capabilities; introduces BenchFill algorithm to effectively reduce local coverage gaps.

Oct 28Wed
  2. Latent Space

    How Anthropic Safely Delivers Top Global LLM: Separating the Model 'Brain' from the Agent 'Hands'

    Anthropic delivers the Claude platform safely and launches Opus 5.5 version through techniques such as separating the model 'brain' from the agent 'hands', expanding APIs, and building reliable hosted agents. The newly introduced S-1 explains improvements and investment cycles, which are crucial for the platform to enhance AI safety and efficiency. Claude Opus 5.5 has been released, including model behavior, evaluation, and pricing descriptions.

Oct 27Tue
  1. Hugging Face Daily Papers65

    Continual Learning Exam (CLE) released: Evaluating LLM agents' long-term continual learning and experience learning benchmarks

    Continual Learning Exam (CLE) released, a benchmark including 120 real tasks, 600 progressive levels, and 89 professions, for evaluating the long-term continual learning and experience learning of LLM agents. Tested 17 cutting-edge configurations, the strongest system Claude Code + Opus 5 only completed 47.7% of the levels and 23.3% of the task steps.

Oct 26Mon
  1. Hacker News · AI hot posts

    Why is AI-generated language becoming harder to understand?

    Research found that text generated by AI models uses language features more frequently associated with understanding difficulty, such as long syntactic dependencies, low-frequency n-grams, and the use of inanimate subjects with active verbs, compared to human writing. Compared to human writing, 9 cutting-edge and near-cutting-edge models use more low-frequency n-grams, most of which also use longer dependencies, more stacked nouns, and more ellipsis of object clauses, with significant differences among models. The study suggests that AI text is optimized for goals such as mathematics, code, and reasoning, leading to information density prioritizing effective communication, and this hypothesis is verified by fine-tuning language models.

  2. Browser Use38

    Web Agent Benchmarks: WebArena, WebVoyager, Mind2Web and More

    The benchmarking methods for AI agents on websites like WebArena, WebVoyager, and Mind2Web differ. WebArena and similar sandbox environments provide reproducible scores, while real website environments like WebVoyager are scored by humans or LLMs. Offline datasets like Mind2Web compare prediction steps with human behavior. Browser Use has released BU Bench V2.1, which uses GPT-5.6 Luna for scoring in real website environments and includes model cost and time data.

Oct 25Sun
  1. The Decoder72

    OpenMobile-2, an open-source smartphone AI model, claims to be comparable to proprietary models.

    Researchers from Nanjing University and institutions like SenseTime have released OpenMobile-2, which includes two AI models based on Qwen3.5 (9B) and Qwen3.6 (27B), and has publicly disclosed the training environment, data, and reproduction instructions.

    Why it matters: Research fixed the training data size while only increasing the number of applications, observing an improvement in success rates for unseen applications, providing a reproducible path for data construction of mobile agents.

Oct 23Fri
Oct 22Thu
  1. Sierra Blog75

    𝜏-Entity: How does the speech intelligence agent accurately capture entity information

    The 𝜏-Entity benchmark speech intelligence agent captures the accuracy of entity information such as names and phone numbers in 200 tasks. GPT Live 1 has a correct rate of 60.3% under normal conditions, but only 34.0% under noise and voice interference; recommended verification strategies can increase success rates by 14-31 percentage points, but call duration increases by 21-28 seconds.

  2. Hugging Face Daily Papers65

    StreamEval: Benchmark for evaluating interactive streaming video generation models

    StreamEval benchmarked the performance of interactive streaming video generation models on five tasks, finding reliable request execution and sustained state consistency to be major challenges. The benchmark includes 300 scripted test cases, covering character dialogue, character performance, world generation, instruction editing, and reference editing tasks, scored across four evaluation dimensions (task execution, state consistency, visual fidelity, and audiovisual coordination).

  3. MarkTechPost55

    Cactus Whistle 16.9 MB Speech Model Performance Evaluation on CPU: Comparison Results with Whisper Base

    The Cactus Whistle speech model transcribes seven languages on the CPU with a 16.9 MB file size and returns word timestamps and probabilities. Tests show it outperforms Whisper Base on its test-other dataset, but there is no significant difference from beam search, with the speed advantage shrinking as audio length increases. Additionally, it tested 30-second window audio processing, streaming latency, timestamps, keyword bias, other languages, noise, embedding, and speech-to-tool calling features.

Oct 21Wed
  1. OpenAI Alignment Blog50

    OpenAI introduces the complementary metric g-mean² to enhance model monitorability evaluation

    OpenAI has introduced the complementary metric g-mean² to improve model monitorability evaluation. g-mean² calculates TPR and TNR through behavioral labels and monitoring predictions, but cannot solve the problems of mixed differences in different model tasks and deployment risks. The new metric includes a paired matching metric for aligned shared instances and a post-control integrated metric simulating monitoring rejection of outputs, to more comprehensively evaluate model deployment risks.

  2. Hugging Face Blog58

    FairStress: A benchmark for a more in-depth evaluation of LLM decision fairness

    FairStress benchmark evaluated LLM performance in demographic fairness across 300 decision scenarios across 29 fields (such as recruitment, healthcare), including 13 million test cases. The study found that the same persuasive prefixes (adversarial prefixes) have varying impacts on model decisions across different demographic groups, and there are differences in how models explain decisions containing demographic information.

Oct 19Mon
  1. Artificial Analysis Articles70

    AA-PersonalityBench releases: a benchmark to measure the communication ability of AI models

    AA-PersonalityBench evaluates 30 personality traits, including friendliness, through 900 questions, using LLM judges for blind pairwise judgments, providing Elo scores and 95% confidence intervals; Claude Sonnet 5.5 scores highest in traits like friendliness (Elo 1000), and Gemini 4 Argon (high) performs outstandingly in traits like transparency (Elo 1059).

  2. Artificial Analysis Articles

    HealthBench and HealthBench Professional: Evaluating AI model performance in real health conversations

    OpenAI has released the HealthBench and HealthBench Professional evaluation benchmarks to assess the performance of AI models in real health conversations. In HealthBench Professional, GPT-6 Astra and GPT-6.1 Sol scored the highest after length adjustment (66.0% and 65.1%), while Claude Sonnet 5.5 scored the lowest (63.0%).