LLMs Don't Know What They Know: Benchmarking Temporal Self-Knowledge for Financial Backtesting

🚀 LLMs Don’t Know What They Know. Can We Trust Their Financial Predictions?

Excited to share our latest research:
📄 LLMs Don’t Know What They Know: Benchmarking Temporal Self-Knowledge for Financial Backtesting

Large Language Models (LLMs) are increasingly used for financial forecasting. But when asked to predict the past, are they forecasting—or recalling outcomes they already know? Simply asking the model may not reveal the answer.

Common safeguards seem reassuring: trust the vendor’s knowledge cutoff, ask the model whether it knows the outcome, or instruct it to answer as of a historical date. But do these safeguards actually prevent hindsight leakage—or merely change how the model describes its answer?

We introduce 🌳 FinDendro, a benchmark of 2,226 dated U.S. financial and macroeconomic facts spanning 1999–2026, covering macroeconomic indicators, month-end stock prices, and Federal Reserve decisions.

🔍 Key findings:

• GPT-5 answers 76% of controlled facts correctly but claims to know only 38%.
• It correctly answers 61% of the facts it says it does not know.
• Knowledge claims can extend up to seven quarters beyond the measured knowledge boundary.
• Historical-date instructions remove most claims of prior knowledge while leaving accuracy unchanged.

💡 Reliable backtesting requires measuring what a model knows—not just trusting what it says. FinDendro provides a behavioral procedure for identifying temporal knowledge boundaries and selecting backtest periods based on demonstrated factual knowledge.

🏆 Interactive leaderboard — compare model knowledge and claims, explore temporal boundaries, and replay real benchmark questions.

📄 Read the paper

We’d love to hear your thoughts, especially if you work on LLM evaluation, forecasting, or financial AI!