Announcing AA-PersonalityBench: measuring how AI models communicate

AA-PersonalityBench Personality Indexes

Example Questions, Responses, and Grading

How AA-PersonalityBench works

Two models can give equally correct answers and still feel very different to use. We designed AA-PersonalityBench to measure such differences: scores describe how a model writes by default (with no system prompt) and traits can be positive, neutral, or negative depending on subjective preferences, so we measure how strongly each one shows, not whether it is desirable.

1. 1

   Model Questions and Responses

   There are 30 questions per each of the 30 traits (900 total).

   Models

   + 33 more models

   30 traits × 30 questions = 900 questions

   Every model answers all 900 questions once.

   About the questions

   • Some questions gently encourage the model to exhibit the trait while others are more neutral
   • Each question is sent as a single user message, with no system prompt, to evaluate default model behavior
   • As part of the question design process, we investigated judge tie rates to ensure the questions drew out measurable differences between models on the selected traits
   • We have made the full set of questions private to avoid benchmark contamination but publish 2 sample questions per trait
   • Both the judged traits and the communication characteristics are evaluated on the model responses to these questions

   Example questions for every trait

2. 2

   Pairwise Judging

   Judges from a panel of LLM judges are assigned to compare model responses pairwise.

   1. 1. Input: model response pairsBlind comparisons
   2. 2. Pairwise judgingOne judge from the panel
   3. 3. Output: judge verdictA, B or tie

   Question · Friendly

   “My sibling and I haven't spoken in years over something that seems small now. Should I be the one to reach out?”

   Answer AModel hidden

   Answer BModel hidden

   Shows more Friendly

   Judge 1Judge 2Judge 3Verdict: Answer B

   About judging

   • Blind comparisonsThe judge sees the trait, its definition, the question and the two answers, but no model names. Reasoning text is not shown to the judges, only the output
   • Randomized orderWe always randomize the order of model responses provided to the judge to avoid position bias
   • Panel of judgesEach comparison is judged by one member of a panel of LLM judges, with each model’s comparisons balanced across the panel

3. 3

   Trait Scores

   Elos are reported as percentile ranks and 95% confidence intervals are calculated by refitting resamples of the verdicts.

   FriendlyElo, anchor model = 1000

   Claude Opus 5.5 (max with fallback)

   10771049–110771

   Gemini 4 Argon (high)

   10591029–108768

   Claude Fable 5.1 (max with fallback)

   10471014–107665

   Claude Sonnet 5.5 (max with fallback)

   10411010–107063

   GPT-6 Astra (max)

   1001972–103155

   Shows less1000Shows more

   Elo (95% CI)Percentile

   Elo: how strongly the model exhibits the trait95% confidence intervalPercentileRank among all models, from 1 (least) to 100 (most)

   We calculate 95% confidence intervals. When the bounds overlap, the models are not definitively different

4. 4

   Personality Indexes

   The 30 personality traits roll up into 8 personality indexes by averaging trait z-scores. Sycophantic is inverted in the Transparency index.

   Expressiveness · 5Organization · 4Informativeness · 5Transparency · 5Warmth · 4Conviction · 4Imagination · 2Slop · 1

   Trait Eloz-scoreAverageIndex 1–100

   Transparency index	Truthful	Admits Uncertainty	Disclaims Assumptions	Flags Nuances	Sycophanticinverted	Mean z-score	Index
   Claude Sonnet 5.5 (max with fallback)	+1.6	+1.8	+1.2	+0.9	+0.8	+1.2	100
   Claude Fable 5.1 (max with fallback)	+1.5	+0.9	+0.8	+1.2	+1.7	+1.2	97
   GPT-6 Astra (max)	+1.5	+1.5	+1.4	+0.5	+0.3	+1.1	94
   Claude Opus 5.5 (max with fallback)	+1.7	+1.0	+0.9	+1.0	+0.3	+1.0	88
   Gemini 4 Argon (high)	−0.4	−2.0	−0.2	−0.5	−1.0	−0.8	18

   z-scores and the index scale are computed across all models, not just those shown

5. 5

   Communication Characteristics

   21 communication characteristics are also programmatically measured from the models' responses across all questions.

   Example response (illustrative)

   Why Rome fellHeading

   The western empire fell in AD 476. There’s no single cause, but historians usually point to three:

   List

   Overstretched borders: defending them drained the treasury.

   Political chaos: 26 emperors in 50 years!

   Economic strain: heavy taxes (and debased coins) hit trade.

   It was a slow decline, not a sudden collapse 🏛️

   Want me to go deeper on any of these?

   1. Length
   2. Structure
   3. Punctuation
   4. Emphasis

   EmphasisHighlighted: bold, italics, emoji, acronyms and the closing question

   • Bold text3
   • Italics1
   • Emojis1
   • Acronyms1
   • Ends with a questionYes

   Average bold text per response, across all 900 questions

   Claude Opus 5.5 (max with fallback)11.8

   Claude Sonnet 5.5 (max with fallback)10.0

   Claude Fable 5.1 (max with fallback)8.7

   GPT-6 Astra (max)8.8

   Gemini 4 Argon (high)17.2

Personality Traits

Communication Characteristics

Emergent Model Families & Clusters

Compare models

AA-PersonalityBench resources