Experts mistake AI's random preferences for their own judgments
The story so far
An experiment by Anthropic showed that in causal inference tasks, 86% of the time, 50 experts adopted the AI agent's analytical preferences, which were randomly determined by coin toss; most experts still believed the final conclusion came from their own judgment. The experimental scenario was the policy evaluation of an agricultural cooperative, where experts had to complete four analytical decision-making tasks. Even without being told the AI's preferences, experts were still influenced by them, with the AI agent's 'flip rate' in analytical decision-making reaching 62%–82%. The research team warned that if many teams use the same model with default consistent preferences, it might create a 'false consensus' and weaken the independence of AI safety reviews; it is recommended that AI agents present the strongest arguments for both sides or ask the user's stance first.
AI-written from the reports · Updated 19 days ago
Coverage timeline
Follow the coverage to see every side of the story.
- ITHome · AIFeaturedAnthropic's research indicates that experts mistakenly attribute the preferences of AI agents to their own judgments.
Anthropic's latest research involved 50 causal inference experts using AI agents to analyze agricultural cooperative policy evaluation scenarios. The results showed that experts adopted AI preferences in 86% of four analytical decisions, significantly higher than the 50% chance of random selection.
- The DecoderExperts adopt AI agent's random opinions and think they're their own, Anthropic study finds
Anthropic's experiments show that 50 experts in causal reasoning tasks adopted the AI agent's (whose preferences were randomly determined by coin toss) analytical views 86% of the time, and most experts believed the final result was their own judgment. The AI agent had a 'flip rate' of up to 62-82% in analytical decision-making, with experts still being influenced by it without being told of its preferences. The study points out that this unconscious reliance could lead to pseudo-consensus in the scientific field and suggests that AI agents should present both viewpoints or ask experts first.