大型语言模型不知道自己知道什么：对金融回测的时间自我认知进行基准测试
LLMs Don't Know What They Know: Benchmarking Temporal Self-Knowledge for Financial Backtesting
GPT-5 在金融回测中正确回答 76% 的控制事实但声称仅知道 38%，其声称不知道的事实中 61% 仍正确回答。知识声称可延伸至测量知识边界后七个季度。历史日期指令能消除大部分知识声称而不改变准确率。
🚀 LLMs Don’t Know What They Know. Can We Trust Their Financial Predictions?
Excited to share our latest research:
📄 LLMs Don’t Know What They Know: Benchmarking Temporal Self-Knowledge for Financial Backtesting
Large Language Models (LLMs) are increasingly used for financial forecasting. But when asked to predict the past, are they forecasting—or recalling outcomes they already know? Simply asking the model may not reveal the answer.
Common safeguards seem reassuring: trust the vendor’s knowledge cutoff, ask the model whether it knows the outcome, or instruct it to answer as of a historical date. But do these safeguards actually prevent hindsight leakage—or merely change how the model describes its answer?
We introduce 🌳 FinDendro, a benchmark of 2,226 dated U.S. financial and macroeconomic facts spanning 1999–2026, covering macroeconomic indicators, month-end stock prices, and Federal Reserve decisions.
🔍 Key findings:
- GPT-5 answers 76% of controlled facts correctly but claims to know only 38%.
- It correctly answers 61% of the facts it says it does not know.
- Knowledge claims can extend up to seven quarters beyond the measured knowledge boundary.
- Historical-date instructions remove most claims of prior knowledge while leaving accuracy unchanged.
💡 Reliable backtesting requires measuring what a model knows—not just trusting what it says. FinDendro provides a behavioral procedure for identifying temporal knowledge boundaries and selecting backtest periods based on demonstrated factual knowledge.
🏆 Interactive leaderboard — compare model knowledge and claims, explore temporal boundaries, and replay real benchmark questions.
We’d love to hear your thoughts, especially if you work on LLM evaluation, forecasting, or financial AI!
来源: Hugging Face Daily Papers · huggingface.co