Blog

This paper announces the results of a large study to evaluate the ability of AI agents to solve unpublished research-level problems in theoretical physics. If you are at all interested in physics, please take a look.

The abstract should be self explanatory. I’ve been thinking about these models quite closely for a while. But, frankly, I wasn’t expecting this day to come as soon as it has. After we found these results, I couldn’t sleep properly for a couple of nights.

Let me add some back story since the timeline is revealing.

About 6 months ago, just for fun, I prompted a frontier agent with an unpublished problem whose answer I had worked out. The agent got the answer wrong but it was impressive that it managed to make any progress at all, and could solve the problem after a few hints.

Praneeth suggested it might be fun to collect a suite of such problems along the lines of the mathematicians’ First Proof assessment. Soon we had a small core group and started requesting experts across the world to contribute problems. Problem collection was a slow process, since we were making a significant ask of contributors. We wanted unpublished problems, with clean verifiable answers. So, our contributors had to either sacrifice potential publication elsewhere, or else undertake a small research project just for us.

Even as problems started coming in, the agents started improving as newer models were released. We kept urging people to push up the difficulty of their problems. But, there is a ceiling to this difficulty since even world experts have only so-many problems lying about whose answers they know and haven’t published.

I don’t know of any individual who can solve any significant fraction of our final collection of problems without AI assistance. For instance, I can solve the problems I contributed myself and understand a few others. Most problems don’t even make sense to me.

Our initial evaluation suggested that GPT-6-Astra’s score was in the late eighties followed closely by Opus 5.5

But, as we started to investigate the mistakes that the models might have made, the models, especially Astra, insisted that they were right and that it was the expert-supplied answers that were erroneous. When we went back to the experts it turned out, in multiple instances, that that the experts found some subtle errors buried deep inside their calculations. As they revised their answers, Astra’s score started to inch up and it finally ended up with 98% (49/50).

The one problem it doesn’t solve is not because it makes a “mistake”. This is a trick-prompt, and requires the model to make an inspired guess, and then bend the rules a little. It can’t do that yet.

We comment a little on the broader implications in our conclusions. Clearly, these agent are enormously powerful tools that will accelerate discovery in the short run. They also raise serious structural concerns about access, equity and pedagogy.

I’m giving this outreach lecture on Sunday morning as part of the Science Gallery’s QUANTUM exhibition. I’ve promised a 9-year old that some of it will be accessible to him. But, if you come to Hebbal, I’ll try and have something of interest for others as well. Register here.

A recording of a colloquium at TIFR. The objective was to advance a discussion we are having across the institute on the potential impact of AI on scientific institutions and, more broadly, on India. The first part is slightly technical, although hopefully still useful for a relatively-broad audience.

▶

My oped today in the Indian Express explaining that India is facing a technological crisis, and proposing a compute tax (in kind) on data centres.

On the arXiv today, with Purvam Jain, Preethi Jyothi and Vihari Piratla. https://arxiv.org/pdf/2605.30788 Modern LLMs have become increasingly good at a variety of languages. How does one detect cross lingual gaps in their abilities? We propose a set of synthetic puzzles that can be generated using base templates and then scaled up dynamically. This avoid translation errors; the tasks are clearly quantifiable; are comparable across languages; and as models improve it is trivial to scale up the test. The flip side is that these tests can’t detect more subtle issues like lack of awareness of nuance, or differences in creative-writing skill. Nevertheless, empirical experiments across many models, and seven languages — English, Hindi, Arabic, Chinese, Japanese, Tamil, Telugu — show that this benchmark can successfully detect cross lingual gaps. The complexity-accuracy graphs that we studied with Praneeth from a theoretical perspective in https://arxiv.org/abs/2601.14175 turned out to be useful. By studying the entire complexity-accuracy curve, we sidestep issues that arise if one tests the model at only a single complexity level.

This is an article I wrote 7 years ago analyzing the allegations against the Iranian nuclear program and the history of the conflict between the US and Iran. Reading it again, I think it is still useful to understand some of the context and history, which is usually glossed over in the media. The article includes a number of references to further reading. World history moves slowly on human timescales. But looking back, some broad trends were quite clear. The last line of the article is definitely one that I would write again today.

https://mronline.org/2019/07/01/imperial-overreach-in-iran/ (Originally written for the Research Unit in Political Economy, but then republished in Monthly Review online.)

Today, with Hao, Andreas, Mark, Lisa and Carlos: https://arxiv.org/abs/2602.06543

We tried to ensure that our main points are clear from the intro, even for a non-expert. I encourage you to click through and read at least the first few pages.

Here is a longish semi-popular summary of some of the background for those who are interested.

While formulating the information paradox, Hawking assumed that observables inside the black hole are independent of those outside. (More precisely, algebras of operators inside and outside the black hole commute with each other.) This assumption is so common that Hawking called it a “basic assumption of quantum theory”. Page disagreed with Hawking’s conclusion and suggested that information would emerge gradually from the black hole, according to what is now called a “Page curve.” But Page’s argument relied on precisely the same assumption of independence of degrees of freedom.

In quantum gravity, this assumption fails. Instead, under weak assumptions, one finds a “principle of holography of information”: observables in the black-hole interior (or any compact region) can be rewritten in terms of observables in the exterior when the global state is pure.

“Holography of information” does not give us a full-fledged holographic duality like AdS/CFT. On the other hand, it applies to asymptotically flat space and does not require us to know details about the UV-complete theory.

This leads to the following picture. Consider surrounding a black hole with a detector. If the detector makes only coarse-grained observations, it naturally loses information, and will see a rising curve for the von Neumann entropy of the exterior. But if the detector makes suitably fine-grained observations, it will find a von Neumann entropy that is constantly zero. The exterior observer knows about the interior at all times — and not just after the “Page time.” So holography of information resolves the information paradox but simultaneously trivializes the Page curve.

This is somewhat different from the historical bias within the hep-th community, which has been that Hawking was wrong but Page was right. And, several recent computations of the Page curve in toy models of black holes seem to vindicate this bias.

In 2021 the G7(=all of us + Sanjit) noted that these models relied crucially on a nongravitational bath. In the presence of this bath, the bulk theory of gravity follows nonstandard dynamics. (The graviton becomes “massive”.) The G7 showed that when gravity was turned on in the bath, the Page curve trivializes. In the presence of a bath, the exterior observer can reconstruct only a part of the black-hole interior called an “island”. But the G7 showed that islands are inconsistent in standard gravity because we cannot have exactly-defined algebras for a compact region when the global state is pure. I’ll call this the “inconsistency paper.”

To be clear: the “inconsistency paper” doesn’t suggest that computations with a bath are wrong. These computations are nice and lead to interesting questions about the gravitational path integral that we are still exploring. But they provide a misleading physical picture for realistic black holes.

Last year, Stefano, Henry, Chang-Han and Geoff wrote an “Apologia for islands” arguing that islands and the Page curve are relevant even in standard gravity. I’ll refer to this as the “apologia paper.”

The “apologia paper” doesn’t raise any technical objections to the “inconsistency paper.”

The “inconsistency paper” relied on the observation that the exterior observer can measure the Hamiltonian in gravity. The “apologia paper” sets up configurations where the Hamiltonian is inaccessible to the exterior observer. For example, in AdS, one can prevent the observer from making measurements on part of the boundary. This is like having a detector with a “blind spot” and moreover, rather than studying a generic black hole, one studies a black hole that is localized right next to the blind spot. In flat space, it is possible to consistently discard the Hamiltonian by hand at null infinity. If the Hamiltonian is inaccessible to the exterior observer, the argument from the “inconsistency paper” obviously doesn’t apply.

These caveats were known even before the “inconsistency paper.” In 2020, when we studied the holography of information with Alok, Pushkal, Siddharth, we discussed a Page curve of this kind in AdS. And we also showed that the Hamiltonian can be discarded from the set of observables at null infinity to get a Page curve.

But these Page curves don’t teach us about information “emerging” from the black hole. They simply tell us about how information is redistributed between the “blind spot” and the rest of the detector. And the islands one gets this way are not the islands we studied in the “inconsistency paper.” Rather than being compact regions, they always have an asymptotic piece corresponding to the “blind spot” in the detector.

This is why our paper today is titled “Seeing Page Curves and Islands with Blinders On.” We have many more details, including a detailed discussion of “relational observables” and why they can’t be used to make islands consistent; and an explanation of how holography of information is important even in the presence of a bath.

I doubt that there are technical disagreements on any of these points. So, I’ll end with some personal perspective. Arguably, the effect that observables outside a compact region are sensitive to observables inside the region is one of the most interesting facts that we have learned about quantum gravity. It explains how information from the interior ends up in the radiation. So I feel that it makes sense to embrace and explore this physics, rather than finding innovative ways of obscuring it, as one must necessarily do to see a Page curve.

If you are in Hyderabad over the weekend, here are a couple of fun events that I’ll be part of on Saturday. A panel discussion at the Hyderabad Lit. Fest and then a public outreach lecture at Lamakaan. Both events are free and open to the public.