理论物理学家如何用 AI 模型解决前沿问题:TheoPhysEval 基准测试揭示 GPT-6 Astra 的惊人能力
I spoke to seven physics experts grappling with the AI revolution
理论物理学家 Suvrat Raju 创建 TheoPhysEval 基准测试,发现 GPT-6 Astra 能独立解决 90% 的专业问题,且平均每题计算成本不足 3 美元;多位顶尖物理学家正利用 Claude、ChatGPT 等模型加速研究,但该领域因依赖实验验证,AI 的实际作用仍存争议。
Today’s article is sponsored by 80,000 Hours, a podcast that explores the transformative impact of AI.
About six months ago, Suvrat Raju, a professor of theoretical physics at the International Centre for Theoretical Sciences in Bengaluru, asked a frontier AI model to solve a physics problem. He didn’t ask because he needed to know the answer. He had actually solved this particular problem already. But he had never published the solution — which made it a perfect question to test how good AI was at physics.
LLMs have been trained on the whole Internet, so if a physics problem has been discussed online, it’s likely the model already knows about it and may have memorized the answer. So a fair test of AI capabilities needs to focus on problems that have never appeared online.
The model didn’t answer Raju’s problem correctly, but it did better than he expected — well enough to get him thinking the experiment should be repeated and formalized. Along with a colleague, he set out to create a theoretical physics benchmark for AI.
They asked other physicists for some of their private gems — minor but real findings they’d never gotten around to cleaning up for public display. They were asking for real work, problems that take humans at the top of the field about a day to solve.
“Our contributors had to either sacrifice potential publication elsewhere, or else undertake a small research project just for us,” Raju wrote.
By the time all the questions and answers were ready, the AIs were too. At a colloquium in September, Raju shared preliminary results showing that leading AI models could solve about 60% of the problems on their own. And by the time he and his colleagues finished the full set, the situation had advanced further: GPT-6 Astra could solve nearly 90% of the problems.
Curious to know where Astra was struggling on the rest, they gave the model the human-written solutions and asked it to figure out where it had gone wrong. To their surprise, Astra argued that it hadn’t made a mistake — the humans had. Astra identified subtle errors in the human-written answers. In reality, Astra had answered 49 of 50 problems correctly. And the one problem Astra missed was something of a trick question.
Each correct answer had cost less than $3 of compute on average — “less than the price of coffee in Paris,” as Raju put it in a talk — and had come after a single human prompt.
In other words, Raju’s TheoPhysEval benchmark was saturated before it was released. When Raju found out, he didn’t sleep well for a couple of nights, out of a hard-to-disentangle blend of astonishment, excitement, and fear.
I asked him if there would be a second version of the benchmark. He said no.

“People gave us the hardest problems they could,” he told me. “We’ve kind of hit the ceiling of well-defined calculations that people can do. Now, there is open research and you can keep asking about open research. But that’s not an evaluation.”
For this story I talked to Raju and half a dozen other theoretical physicists. All described how AI was beginning to transform their field. Like programmers and mathematicians, they and their colleagues are increasingly using large language models in their work. This has dramatically accelerated the pace of calculation and research for many, leading to an explosion of new papers and awkward questions about how to train the next generation of physicists.
But physics is different from math in important ways and may not follow the same path. It is bound to the real world through experiment. And its most open questions are so open that it is unclear how quickly AI can help. It is even possible that AI will, at least for a while, allow physicists to calculate less and focus more on old-fashioned and more conceptual ways of working.
AI is rapidly changing physics
Raju isn’t the only physicist to notice that AI models are improving rapidly. In October, a Turkish physicist posted on X that Claude Opus 5.5 had recreated the core of his PhD thesis. It had taken him a year and 500 pages of hand calculations. Claude reproduced his work in 10 hours.
Ed Witten, the most-cited living theorist, has been crediting Opus in recent papers. Juan Maldacena, another of the field’s eminent figures, acknowledged help from OpenAI models on two papers this year and told me he uses it “to do some calculations, search the literature” and that increasingly many physicists find AI helpful with “calculations that in the past were maybe too long.”
Andreas Karch, a physicist at the University of Texas at Austin, is running AI agents around the clock. “It’s pretty much running 24 hours a day. I always have it do something,” he said.
Karch ruefully admitted that he’d written one of the subtly wrong answers in Raju’s benchmark. Now he sends his old papers through Astra, checking for missed errors. “I do one paper a day now. Pretty much every paper that I send it, it has something to say.”
Karch gets a high-tier ChatGPT subscription for free from OpenAI, which likes to see physicists using its models.
Harvard’s Matthew Schwartz uses AI agents even more aggressively. During an October interview, he paused to count the number of sessions running on his machine.
“Nine, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20,” he told me. Each of these threads was working on a different problem, with 10 to 20 agents per thread.
Schwartz isn’t paying for all of those tokens. He told me that Anthropic recently asked him, “‘Suppose you had more tokens, what would you do?’ And I said, ‘I don’t know, let me try.’ And so they just gave me tokens. And now I’m seeing what you can do with tokenmaxxing. There’s a whole bunch of physicists who do this. They have contracts with a bunch of academics.” Schwartz has written a couple blog posts for Anthropic about his AI and physics work.
Of course, this isn’t the whole story. These agents are mainly performing calculations, and not all theoretical physicists spend their time calculating. Daniel Harlow at MIT said that in his corner of physics, “the vast majority of the work is figuring out what to do.” By the time he has a calculation to perform, he said, “I’m like 95% done already.” Instead, he finds LLMs useful for catching up on farther-flung corners of physics and learning borrowable ideas from adjacent fields — nicely helpful but not a category change in his work.
A flood of papers
The publication of theoretical physics papers has soared in recent months. From January to October, physicists submitted about 30% more papers to the hep-th category of the science repository arXiv — hep-th is short for high-energy physics theory — than in the same period last year. In the prior two decades, the highest annual jump was 8%.

The obvious question is: are all these papers slop? The answer is yes and no. Schwartz described many of them as “poorly written, but I think the science is probably pretty solid.”
But nobody knows for sure because nobody has time to read all the papers. “It’s like this endless buffet. Even if it’s the most amazing food, how much of it can you digest?” is how Scott Aaronson put it to me. Aaronson is a theoretical computer scientist who has helped draw important connections between computer science and physics over the last 15 years.
Eventually, and perhaps soon, the endless buffet will require some new ways of organizing how physics as a field operates. Syracuse University physicist Aron Wall thinks it will increase the necessity of personal references and relationships. “My source for papers on the arXiv usually is somebody tells me about them, and if they don’t, then I don’t need to know,” he said cheekily. (Wall consults for Anthropic.)

Maldacena, for his part, isn’t sure what the solution might be, but he thinks it’s not an insurmountable problem. “It’s hard to process. But I think it’s great, and we’ll learn how to do that,” he said.
An uncertain future for young physicists
The flood of papers may be the kind of manageable change science has always had to deal with. But AI could have more profound impacts on the career prospects of young physicists.
Established physicists like Maldacena probably don’t have much to worry about. They’ll likely be able to keep their tenured academic jobs in any event. But physicists in training may not be so lucky.
Until recently, performing calculations was a rite of passage. It allowed young people to build both a reputation as a physicist and an intuition for how to approach the deepest problems.
“It’s how we judge students, how we decide students have talent,” Raju said. “I went into physics because I participated in the International Physics Olympiad, and that was considered to be a sign of having talent.”
“If you do enough of this mechanical grinding through problems, at some point, you pick up the concepts behind it,” Karch said.
But Karch said that trajectory may now be changing: “It’s not clear how you jump over that step” if AI is performing all the routine calculations.
“A lot of them feel like that is being taken away from them,” Aaronson said.
Karch also mentioned a burgeoning fear of getting scooped. “If you tell somebody, they’re going to run and ask the agent to solve it,” he said. “You have to be much more guarded if you think you have some big idea.”
Raju told me he’s not taking new PhD students until he gets a better sense of what their futures might look like.
All this is compounded by funding cuts the Trump administration is making to science, which were already threatening the field’s inherently tenuous career path. And while many physics skills translate to AI — Anthropic and OpenAI have both had physicists in high positions — the frontier companies aren’t hiring great numbers of them these days, especially outside top departments like Stanford or Berkeley.
Some students are taking on gigs writing training problems for data companies like Mercor or Scale AI for $80 to $100 an hour. “This is good money on the side, but it’s not a career,” Karch said. “No health insurance comes with this.”
Physics might be just months behind mathematics
As far as anyone can tell, the flood of physics papers doesn’t contain any major AI-aided physics discoveries. The closest thing so far has been OpenAI’s October announcement that one of its models had proved that the math behind the gluons that carry the strong nuclear force holds together in four dimensions. (Brian Greene has a helpful explanation here.) A panel of physicists and mathematicians examined a Lean version of the 1,147-page proof and concluded it is correct. This is half of a Millennium Prize solution and a triumph of mathematical physics that has eluded humans for decades: the theory, called pure Yang-Mills, was first proposed in the middle of the 20th century. But the consensus among physicists seems to be that, technical impressiveness granted, at heart this solution firms up something everybody already knew.
“Existence of Yang Mills? It’s just tying up a [loose] end for the hyper mathematically minded. I don’t think any physicist gives af,” Karch tweeted.
At the same time, nobody I spoke to was confident enough to rule out major discoveries from AI happening soon. They’ve all seen what just happened with math.1
Aaronson told me some of the physicists he knows tend to guess physics is about six to 12 months behind math.
“The reason why this has come for math and for programming earlier than for other things is just that we have very, very self-contained ways to verify whether an answer is right or not,” he said. “And that is what AI needs to help to power its training process.”

Or as Harlow put it, “nobody can just give you a Lean certificate that says, here’s the theory of quantum gravity. So in some sense, that’s the hardest kind of thing for one of these things to try to do because it’s so vague and poorly defined. The rules of the game are completely unclear.”
But Schwartz expects this situation to be temporary. “It’s certain by next summer, the AI will be able to do most of those creative steps on its own and really do papers end to end. And then the summer after that, we won’t even understand what it’s doing,” he said.
AI industry insiders seem to think the same. Last month, the prominent OpenAI researcher known as roon tweeted “unified field theory / quantum gravity within the next 12 months?”
“The problem isn’t the ideas, it’s the experiments”
When Harlow and roon mentioned quantum gravity, they were referring to the deepest open problem in all of physics.
The first half of the 20th century delivered the two pillars of contemporary physics: general relativity, which essentially describes the force of gravity, and quantum mechanics, which essentially describes the three other fundamental forces (electromagnetism and the weak and strong nuclear forces).
The lion’s share of the work in the second half of the century was devoted to fleshing out the second theory. This resulted in what’s called the Standard Model of particle physics, which lays out the properties of all known fundamental particles that make up the universe. The Standard Model was essentially completed in 2012, when the final major particle it predicted, the Higgs boson, was discovered at the Large Hadron Collider.
But those two central theories of general relativity and quantum mechanics not only don’t match up everywhere, they conflict in places. There is no confirmed theory of the physical world that encompasses everything physicists know. This is where quantum gravity comes in. It’s a placeholder term — the name of a theory that doesn’t really exist yet.
The closest candidate is string theory. It does contain both quantum mechanics and gravity, which is why it attracted a lot of excitement when it was developed a few decades ago. It also, among other issues, allows for so many possible universes, 10500 by one common estimate, that nobody knows if our particular universe is among them.
One reason we don’t know is that we can’t test string theory.

The Large Hadron Collider is the most powerful particle accelerator physicists have constructed to date, but it would take a machine roughly a quadrillion times more powerful to make gravity’s quantum effects strong enough to directly observe; a collider as powerful as the LHC would need a radius measured in the hundreds of light-years. Indirect, non-collider tests have been proposed, but so far nothing significant has panned out.
As Karch put it, “the problem isn’t the ideas, it’s the experiments.”
This means theoretical physicists interested in quantum gravity have faced a difficult choice. Some have moved on to other problems. Others have made significant — if publicly underappreciated — advances using little more than mathematical consistency and human intuition as their guides.
As I wrote last year in the Wall Street Journal, this project has been controversial in some quarters, unfortunately so, since I and many others consider it to be an enormous feat of human intellectual effort and creativity. Several of the physicists I quoted in this story have worked on what’s called the black hole information paradox — a central front in the clash between quantum mechanics and general relativity.
There is an ironic possibility looming. Suppose AI improves in physics as it has in math, going from talented student to somewhat incomprehensible master very quickly. Maybe in a year or two an AI agent will generate a brilliant solution to quantum gravity that is consistent with what we know about our universe. But unless technology changes, testing such a solution will remain impossible.
The end of “shut up and calculate”
Some of the most impressive physics theorists were never known for their calculating prowess. Take Einstein in the past, or the likes of Leonard Susskind today.
In the most human-friendly case for the near future, the job of theoretical physicist returns to something like Einstein’s famous method, Gedankenexperiment, thought experiment, the act of pressing on contradictions until they break.
Quantum gravity, for instance, is a question requiring extraordinary conceptual imagination as well as intensive mathematical ability. There are few conceptual guides and, compared to many areas of physics, relatively few clearly defined questions to calculate. One of Maldacena’s central innovations, expressed in one of the most cited theory papers ever, hints at the idea that everything in our three-dimensional world might be described or encoded on a two-dimensional boundary far away.
Wall said this way of working requires a certain amount of something other than raw intelligence. “Science doesn’t just progress through smartness, it also progresses through stupidity in a crucial way,” he said. “Stupid means you’re getting the question wrong, and you have to go back and figure out why you’re getting it wrong.”
Many of the physicists I spoke to feel that moving toward an emphasis on concept-work like this and away from human calculation might not be such a loss. “I feel like that style of doing physics and math is probably done, or will be soon,” Harlow said. “But I don’t really mourn that too much, to be honest.”
“For a long time we’ve been told to ‘shut up and calculate.’ [Maybe AI] frees us up to stop and think, and it lets us think about ambitious problems that we couldn’t think of earlier,” Raju said in a recent talk. “Shut up and calculate” is the cliché expression of a widespread mid-century attitude toward quantum mechanics: nobody understands what the theory really means about the world, but the equations work impeccably, so do the math and don’t worry about the rest.
Raju thinks this relatively cheery paradigm will hold for the next few months at least. Past that, he doesn’t know.
If it doesn’t hold, one possibility is that theoretical physics, long a relatively individual field, starts to look more like experimental physics, which is much more collective. If progress on important questions depends on burning huge amounts of compute, physicists will pool resources together, apply jointly for grants from the government and large institutions, and collectively judge the most important questions to ask the AI. Just a few weeks ago, 51 physicists authored a proposal to this effect.
To Schwartz, whatever structure it takes, the promise of AI and physics is thrilling. “I am excited for the science. I think fundamentally most scientists are interested in answers,” he said. Maldacena concurred. “I’m more excited than concerned,” he said. “That means that we have lots of possibilities and promise for the future.”
Raju was more circumspect, but said it’ll be hard to complain about AI-driven discoveries: “If somebody tells us the nature of dark matter, then I suppose we have to welcome it. We can’t say, ‘Oh, we didn’t want to know it so fast.’”
The trade
If AI meant the end of human physics as we know it, but we got a solution to quantum gravity or other discoveries of similar size, would that be a worthwhile trade? “Making a deal with the devil — sell your soul for the answer?” Schwartz asked jokingly. “Yeah, I would. I think you just have to be a little bit more humble.”
Maldacena, who has done as much as anyone alive to probe the boundaries of human knowledge of the universe, said, “I think it doesn’t matter who generates the knowledge,” as long as we can still understand it.
Harlow, by contrast, wouldn’t take that trade. “No. And I probably wouldn’t even read the result,” he said. “I’d just be off playing piano in a bar somewhere or something.”
In this scenario, theoretical physics might start to look something like a humanities field. Handed-down truths? Unprovable or unintelligible knowledge-objects? That sounds like biblical studies or art criticism.
There would still be physicists in this world. On paper, at least, one of the chief jobs of a physicist is to teach physics to other humans. But they would not be discovering knowledge so much as interpreting it. Indeed, it can be argued that math is already entering this state.
This sort of interpretation has been part of physics for a long time, though perhaps not as much as it should have been. Take quantum mechanics. The predictive power of the equations of quantum mechanics is beyond dispute. At the same time, any real understanding of what quantum mechanics means about the world is wildly disagreed upon and even sorely understudied, as the physicist Sean Carroll frequently argues.
Wall was relatively sanguine about this direction.
“People have this idea of physics and mathematics as something where it’s all about just answering the question, making discoveries,” he told me. “But it’s the most common thing in the world for the paper that discovers a thing to be completely incomprehensible to other people because it’s expressed in that person’s language, and maybe the person who translates it into language other physicists can understand gets as much credit sometimes. And that’s actually fair, because that’s an equally important job.”2
Indeed, we’re starting to see AI discoveries in mathematics that could have significant implications for theoretical physics. One of them relates to a major problem involving black holes. In 2012 a few physicists made an alarming discovery: quantum mechanics seemed to demand that a literal wall of fire sit at the event horizon of a black hole — an absurdity that would violate general relativity. Harlow and a colleague figured out a way of saving the situation, coming up with an elegant loophole of a sort physicists are partial to. They didn’t directly disprove the “firewall paradox.” They simply pointed out that based on our understanding of how information works, testing the paradox would take so long the black hole would evaporate first, and so the question was moot. But as Aaronson pointed out to me, one of the 722 mathematics papers OpenAI released on October 6 looks like it shows that if one of computer science’s biggest unproven conjectures, a cousin of P versus NP, turned out to be false, testing the paradox would become more feasible. This leaves Harlow and his colleague’s defense of spacetime as we know it on somewhat weaker — though in all likelihood still safe — terrain.
This is something of a reversal of the science-like ways humanities fields have tried to position themselves in the last few decades, and not to enormous success. As the political theorist Samuel Goldman pointed out recently, “The conflation of scholarship with ‘creating new knowledge’ has been a disaster for the humanities and social sciences.”
来源: Understanding AI · understandingai.org