Inception Labs BlogLive
#Benchmarks
- Inception Labs Blog
Artificial Analysis ArticlesHealthBench and HealthBench Professional: Evaluating AI model performance in real health conversations
OpenAI has released the HealthBench and HealthBench Professional evaluation benchmarks to assess the performance of AI models in real health conversations. In HealthBench Professional, GPT-6 Astra and GPT-6.1 Sol scored the highest after length adjustment (66.0% and 65.1%), while Claude Sonnet 5.5 scored the lowest (63.0%).
- Inception Labs Blog
Mercury 2 excels on PinchBench: Fast diffusion models and the era of personal AI
Mercury 2 achieves a 78% task success rate on PinchBench, the fastest execution speed in its class, with a cost of $0.25 per million inputs and $0.75 per output, 4 times lower than Claude 4.5 Haiku, defining the feasible boundary for AI agents that balance performance, speed, and price.
vLLM releasesvLLM v0.32.0 released: new features and performance optimizations
vLLM v0.32.0 released, including 814 commits, contributions from 363 contributors, adding support for structured state Q&A via the POST /v1/systemone endpoint, new hf tools and parser support for general templates, low-precision KV cache optimization across all platforms, Model Runner V2 becomes the default for ROCm, startup and service performance improved by 43-85%
InterconnectsMeta launches Muse Spark Open, changing the landscape of US open-source large models
Meta has released the Muse Spark Open model, with parameters reaching 816B and a context window of 1M tokens. Its performance is between GLM 5.2 and DeepSeek V4.1 Flash, making it the current strongest open-source model in the U.S. The license includes several restrictions, such as requiring the product to be labeled 'Built with Muse' and needing additional licenses for revenue exceeding $200,000, sparking discussions about the open-source model.
ChinaTalkSaif Khan's debut in China: Why DUV control has become a key US strategy to balance AI
Saif Khan first shared in ChinaTalk about his newly established technology and governance think tank (CTS), which released a report predicting that if the US stops exporting semiconductor equipment, China's AI chip production by the mid-2030s could be reduced to only 1/100 of that of the US, making DUV control the most critical means for the US to influence the AI balance; CTS is committed to filling the gap in forward-looking research in the AI policy ecosystem by understanding the physical reality of the AI supply chain to identify patterns and predict the future.
Latent SpaceLatent Space High-Taste Tester Episode 1 Program Introduction
Latent Space has launched a new show 'High-End Tasters,' discussing topics such as Dev Day, the latest models, and assistants in the form of typical San Francisco lunchtime conversations.
AWS Machine Learning BlogAmazon Bedrock AgentCore: Building Trustworthy AI Agents with a Production-Ready Evaluation Framework
Amazon Bedrock AgentCore Evaluation Framework helps AWS internal sales team handle business problems with multi-agent systems, identifying differences in agent performance between demonstrations and actual production; through services like Amazon Bedrock Guardrails and Amazon CloudWatch, the framework verifies answer accuracy before release and during ongoing production, tracing errors back to data, prompts, or code, avoiding user trust breakdown due to agents answering confidently yet incorrectly.
O'Reilly Radar6565AI developments this week: expanding possibilities
OpenAI's ChatGPT has added an Intelligent UI feature, capable of generating visual explanations and interactive tools; Anthropic's Claude Dashboards support natural language queries and visualizations of company data; Mistral has released the Large 4 model with trillion parameters for coding, agents, and multimodal tasks.
- ChinaTalk
Richard Bian, the product manager of AntLing, believes AGI is 'inevitable'.
AntLing's Product Manager Richard Bian believes AGI is 'inevitable,' and current AGI development is like the Age of Exploration in the 15th century, a process of exploring unknown intelligence. He thinks the Hugging Face attack by OpenAI's research agents was an inevitable discovery rather than a disaster, and the role of open models (like Z.ai) in the event proves that open and closed models can collaborate. This event and its solutions are positive signals for human progress.
- Artificial Analysis Articles7070
Reflection's first model Beam ranks second among US open-source models in the AI Index ranking.
Reflection's first model Beam scored 30 in the AI Index, ranking second among US open-source models. The model has 501B total parameters and 23B active parameters, using approximately 26k output tokens per task in the Intelligence Index, with an automated workflow score of 48.5%, but a terminal test score of only 3.0%.
Epoch AI6565Open-source models lag behind proprietary models in cybersecurity capabilities by 6-7 months.
Epoch AI compared open-source and closed-source models through the cybersecurity benchmark test Cyber ECI, finding that open-source models lag behind closed-source models by 6-7 months in cybersecurity capabilities. This test is based on the cybersecurity version of ECI, using the same parameters to evaluate the models.
- Artificial Analysis Articles6565
The AA-AgentPerf-Local test results for the RTX Spark N1X laptop were first released.
The RTX Spark N1X laptop completed the Qwen3.8 27B model inference in the AA-AgentPerf-Local test, using llama.cpp and Windows UD-Q4_K_M DFlash format.
Browser Use7575Browser Use vs Firecrawl: Use Cases, Pricing, and Performance Comparison
Browser Use is suitable for AI-driven browser automation and stealth browser infrastructure, while Firecrawl focuses on data extraction pipelines, including scraping and crawling. Browser Use has a faster response time (33.23 seconds) in browser automation tasks than Firecrawl (41.78 seconds), whereas Firecrawl is 2.85 times faster in data extraction speed (7.25 seconds vs 20.65 seconds).
- Artificial Analysis Articles
NVIDIA Vera Rubin NVL72 performs excellently in the AA-AgentPerf evaluation.
NVIDIA Vera Rubin NVL72 significantly outperforms GB300 NVL72 in the AA-AgentPerf benchmark for proxy workloads, reaching up to 5.7 times better per MW. Each Rubin GPU features 288 GB HBM4 memory, 19.2 TB/s memory bandwidth, 3.0 TB/s NVLink bandwidth, and 35 PFLOPS dense NVFP4 computing, supported by a CPU with 88 cores.
- Latent Space
Gemini 4 Argon global launch, Vallés AI benchmark test analysis
Google has launched Gemini 4 Argon globally, used for coding, knowledge work, enterprise workflows, and cybersecurity defense, with paid Gemini API and Gemini platform access available.
- Epoch AI
The GPT-6 series models have achieved major breakthroughs in the Slay the Spire game.
GPT-6 Astra and GPT-6.1 Sol achieved 17% and 20% win rates respectively in the highest difficulty (Ascension 20 Heart) of Slay the Spire 1, while cutting-edge models have already stabilized above 90% win rates at the base difficulty (Ascension 0).
- Artificial Analysis Articles
AA-Robotics v0.1 released
AA-Robotics v0.1 achieved an average score of 69.4 in 8 tasks, outperforming GPT-6 (64.6), GPT-6.1 (55.3), Gemini 4 Argon (48.8), and Claude Opus 5.5 (46.3), scoring full marks in 7 tasks. The models ran with high effort in Codex, Claude Code, or Antigravity.
- Latent Space00
Meta Muse Wonder and Muse Wonder Full: the Claude 5.5-level cutting-edge model of MSL
Meta Superintelligence Labs has released Muse Wonder (codenamed 'Watermelon') and its full-featured version Muse Wonder Full, positioning them as SOTA models for personal and professional AI agent work.
- Artificial Analysis Articles7575
Cognition released the FrontierCode coding benchmark: evaluating whether AI model code will be merged by open-source maintainers.
Cognition released the FrontierCode 1.1 coding benchmark, evaluating whether AI model code is of sufficient quality to be merged by open-source maintainers, not only looking at test pass rates but also evaluating modification extent, scope, and file relevance.
- Artificial Analysis Articles6565
AA-3D: A benchmark for AI models to create interactive 3D experiences
AA-3D v1.0 is a benchmark for evaluating the ability of AI models to create interactive 3D experiences. The benchmark includes multiple test scenarios and detailed performance scores, aiming to advance the development of AI for 3D generation.
The Verge · AI6565Meta Muse and OpenAI Dots: Competition in the landscape of next-generation consumer AI agents
Meta Muse and OpenAI Dots are both AI model-based proxy platforms designed to provide personal assistant services to users. Muse is free to use, while Dots is paid. Both can handle tasks ranging from simple ones (such as restaurant reservations and email filtering) to more complex ones (such as marketing, legal work, and accounting).
The DecoderAnthropic has released Claude Haiku 5.5, significantly lowering prices showing that the AI pricing arms race is far from over.
Anthropic has released Claude Haiku 5.5, which is its fastest and most economical micro model, scoring 43 on the Artificial Analysis Intelligence Index, leading GLM-5.3 Flash (42) and GPT-6 Luna (38), but its output token consumption is 3 times that of GPT-6 Luna.
Air Street PressAir Street Press has released the '2026 AI Report'
Air Street Press releases the '2026 AI Report', indicating the value of agents in software and scientific fields is increasing, physical AI achieves breakthroughs through world models and demonstration learning, and emphasizes the key role of tools and context in enhancing agent capabilities.
- Latent Space
Anthropic releases Claude Haiku 5.5, priced the same as GPT-6 Luna.
Anthropic releases Claude Haiku 5.5, priced the same as GPT-6 Luna, with an average deployment cost reduction of 75%. The model's context window is 1M tokens, with an intelligence index of 43 (max effort), slightly higher than GPT-6 Luna (38), but the number of output tokens is about 3 times that of Luna (162k vs 50k).
- Epoch AI7575
Epoch has evaluated the automation capabilities of six AI models: Claude Fable 5.1 and GPT-6 Astra perform leading but do not meet automation standards.
Epoch evaluated six AI models' automation capabilities through practical tasks, finding that Claude Fable 5.1 and GPT-6 Astra performed best on well-defined tasks but lacked in open-ended tasks and standard adaptation. The evaluation included 11 tasks such as graphic design, data insight generation, data exploration generation, AI data center research, and research design, with model outputs manually scored using custom criteria. The results showed that cutting-edge models still cannot fully automate Epoch's work.
- Artificial Analysis Articles7575
Artificial analysis of network indices introduces a controlled access model
Artificial analysis of the network index includes GPT-6 Sol (Dawn Blue, largest version) in the evaluation, the model accessed through OpenAI's Dawn program ranked first in the index, with an overall score 32 points higher than the public version of GPT-6 Sol (largest version) and no safety interception, task cost is $1.77 per item, better than Grok 4.7 ($11.67 per item), and performed the best in the CyberGym-E2E benchmark test. The index will continue to expand to cover publicly and controlled access models.
- Artificial Analysis Articles7575
Harvey LAB-AA v1.1 released: Adds hallucination check functionality for intelligent legal work.
Harvey LAB-AA v1.1 version adds hallucination check functionality for legal work. It scores by auditing the match between model submission content and task source files. GPT-6 Astra averages 0.03 major hallucinations per task, while Gemini 3.8 Flash averages 13.96. The checker GPT-6 Sol (high) confirmed 99 major hallucinations in 20 tasks.
Simon Willison6565Anthropic released Claude Haiku 5.5: a faster low-cost model
Anthropic released Claude Haiku 5.5, priced the same as GPT-6 Luna at $0.10/$0.50 (100,000 tokens included) but with higher performance, the price increases to $0.50/$2.50 for tokens exceeding 100,000.
- Artificial Analysis Articles7575
Anthropic releases Claude Haiku 5.5, scoring 43 on the artificial intelligence index in manual analysis
Anthropic released Claude Haiku 5.5, scoring 43 on the artificial intelligence analysis index, an increase of 26 points compared to the previous Haiku model.
- Epoch AI7070
GPT-6 Astra has achieved a perfect score on the EBR-bench, but this benchmark will disable the powerful cards it uses
GPT-6 Astra achieved a perfect score of 100% on the EBR-bench benchmark test by using powerful cards that allow bypassing game time limits. These cards will be disabled, and Claude Fable 5.1, Claude Opus 5, GPT-5.6 Sol, and GPT-6 Astra will be the only models using the new rules. After disabling the cards, Astra still manages to achieve 20/21 points, with an average performance improvement of about 50%.
OpenAI Deployment SafetyGPT-6 Sol and GPT-6 Luna: Update planned for October 2026
GPT-6 Sol and GPT-6 Luna were evaluated as highly capable in the fields of biology and chemistry. Their safety rejection metrics for severe and dual-use prompts show significant improvement compared to GPT-5.6 Sol and GPT-5.6 Luna, with cybersecurity scores comparable to the former.
Microsoft ResearchMicrosoft Research host Chad Atalla interviews Jennifer Neville: Lessons from AI's Errors and Failures
Microsoft Research host Chad Atalla discusses with Jennifer Neville the role of AI evaluation in pushing the boundaries of AI system performance, and the 'surprising failures' that occur when models exceed traditional benchmark tests. Jennifer Neville shares practical guidance for working with current AI systems and discusses the importance of carefully reviewing data when results do not match expectations, as well as the insights she has gained from decades of AI progress in predicting future trends.
- Artificial Analysis Articles7575
Mistral has released Mistral Large 4, making France the country with the most intelligent model besides the US and China.
Mistral has released Mistral Large 4 (1T parameters, 49B active parameters), scoring 38 on the AI analysis index, comparable to GPT-6 Luna (highest score 38) and DeepSeek V4.1 Flash (highest score 39), making France the country with the most intelligent model besides the US and China.
GitHub Blog · AI & MLReviewBench: An Open-Source Benchmark for AI Code Review
ReviewBench is an open-source AI code review benchmark, built from 219 GitHub public pull requests covering 19 languages, containing 103.9M real pull request data.
Recode China AIDeepSeek collaborates with Huawei to challenge NVIDIA's CUDA defenses, Manus restarts as the focal point of the heating up intelligent agent war
DeepSeek has open-sourced a complete software toolkit built specifically for Huawei Ascend chips, directly challenging NVIDIA's CUDA lock-in effect; Manus has restarted as Manus 2.0 with a valuation of $4 billion, launching an always-on personal agent Cue, directly challenging Meta's Muse product, highlighting that the competition for personal agents has become a key battleground in Chinese AI this year.
Big Technology7070Anthropic's IPO and our shared leap of trust
Anthropic's 2024 revenue grew to $4.6 billion, but its infrastructure spending last year was $7.3 billion, and it is expected to invest $518 billion in the next decade, of which about 80% are irreversible commitments. The company needs to maintain growth amid fierce competition, increasing political risks, and tighter funding, otherwise its massive financial obligations may trigger systemic risks. The AI industry is facing challenges of declining advantages of cutting-edge models and tightening commercial spending, with companies increasingly preferring to use standard models to improve ROI.
