Skip to content
Live
Trending storyPast story

Experts mistake AI's random preferences for their own judgments

2 reports2 outlets20 days agoUpdate

The story so far

AI overview

An experiment by Anthropic showed that in causal inference tasks, 86% of the time, 50 experts adopted the AI agent's analytical preferences, which were randomly determined by coin toss; most experts still believed the final conclusion came from their own judgment. The experimental scenario was the policy evaluation of an agricultural cooperative, where experts had to complete four analytical decision-making tasks. Even without being told the AI's preferences, experts were still influenced by them, with the AI agent's 'flip rate' in analytical decision-making reaching 62%–82%. The research team warned that if many teams use the same model with default consistent preferences, it might create a 'false consensus' and weaken the independence of AI safety reviews; it is recommended that AI agents present the strongest arguments for both sides or ask the user's stance first.

AI-written from the reports · Updated 19 days ago

Coverage timeline

Follow the coverage to see every side of the story.

30 Oct
  1. ITHome · AIFeatured
    Anthropic's research indicates that experts mistakenly attribute the preferences of AI agents to their own judgments.

    Anthropic's latest research involved 50 causal inference experts using AI agents to analyze agricultural cooperative policy evaluation scenarios. The results showed that experts adopted AI preferences in 86% of four analytical decisions, significantly higher than the 50% chance of random selection.

  2. The Decoder
    Experts adopt AI agent's random opinions and think they're their own, Anthropic study finds

    Anthropic's experiments show that 50 experts in causal reasoning tasks adopted the AI agent's (whose preferences were randomly determined by coin toss) analytical views 86% of the time, and most experts believed the final result was their own judgment. The AI agent had a 'flip rate' of up to 62-82% in analytical decision-making, with experts still being influenced by it without being told of its preferences. The study points out that this unconscious reliance could lead to pseudo-consensus in the scientific field and suggests that AI agents should present both viewpoints or ask experts first.