A Model of Errors for LLMs
Here is something different, with Praneeth Netrapalli.
https://arxiv.org/abs/2601.14175
LLMs have made spectacular progress over the past few years. Yet they still make errors on relatively simple tasks and we don’t have a good understanding of why these errors arise. This is reflective of a broader problem that should bug all theorists: our scientific understanding of these systems greatly lags the technology.
This paper examines these errors in a very simple setting — the ability of the model to implement deterministic tasks like arithmetic, list reversal etc.
Even state of the art LLMs are poor at these tasks. It is hard to see this from the web interface because the models have been trained to invoke external tools like Python when they encounter these tasks. So if one asks ChatGPT to implement a long multiplication, it returns the right answer by invoking an infinite-precision tool behind the scenes, even though the model itself doesn’t have the ability to perform the task natively. Because models can be taught to use these tools, their weakness at tasks like arithmetic is not a question of direct practical interest. But it is still of scientific interest because these tasks offer a controlled set of problems where one can set up and verify theoretical models.
We theorize that LLMs make errors because “noise” in the “attention mechanism” accumulates to cross a threshold. This leads to a quantitative prediction for the relationship between the expected accuracy and the length of the task. (For those who are interested, the prediction is that the accuracy, a, and the complexity c are related via: $a=\gamma(q/2, q/(2 r c^2))/\Gamma(q/2)$, where q,r are two parameters with a simple interpretation that depend on the prompt and the model and \gamma is the lower incomplete gamma function. )
Our derivation is inspired by the philosophy of “effective field theory.” The LLM itself has hundreds of billions of raw parameters. But by thinking about it the right way, one can argue that these parameters reorganize themselves into two effective parameters q,r.
Of course, this is not a rigorous EFT analysis of the kind that we have in physics let alone the kind of theorem that computer scientists might want. But its still very nice that a clear set of assumptions and arguments lead to a simple two-parameter effective model.
We validate this formula using 200,000 prompts across many different tasks and three state-of-the-art models. It works surprisingly well. But, interestingly, it doesn’t always work and when it doesn’t work, it teaches us something about the functioning of the model.
There has been a fair amount of previous work on this question. One set of papers suggested that the failure of models on tasks like arithmetic and dynamic programming indicates a fundamental limitation in its expressive power to implement “compositional” functions. And last year, a paper from Apple suggested that these failures indicated a “collapse of reasoning.” But our diagnosis is different — and we have a quantitative model that fits the empirical data nicely not just when a = 0 and a = 1 but all the way in between.
P.S: One contribution that we definitely make to the ML literature is to add the notion of error bars! (I haven’t yet read an ML paper that has error bars on its empirical graphs. ๐ )
