Measuring Diminishing Returns to LLM Intelligence

Evidence from Arena AI preference data

This essay is related to ongoing research with Stella Si, Giorgos Zervas, Mert Demirer, Aaron Kaye, and Nadav Tadelis. This analysis is vibe-coded and should be considered preliminary. More rigorous work on this topic is in progress.

Epistemic status: Possible.

A pivotal question for the AI industry is whether frontier models can sustain a large price premium over open-weight and other cheaper models. If they can, frontier labs may capture much of the industry’s profits. If models at different capability levels are close substitutes, more of the value may instead accrue to other parts of the AI stack.

The answer depends partly on whether users experience diminishing returns to model capability. If each additional increment of capability produces a smaller improvement in user value, willingness to pay for frontier models will also rise more slowly. That would compress frontier-lab margins and weaken the incentive for increasingly expensive training runs.

While many have speculated about competition between LLMs at different capability levels, few have tried to measure it. I present preliminary descriptive evidence consistent with diminishing returns to measured model capability among consumer users. The results suggest that frontier labs may ultimately need to earn rents from products and complementary services, not solely from access to the most capable model. I end by considering what those services might be.

Measuring intelligence

There are many definitions of intelligence. Some are so broad as to include nearly anything and imply that the maximum level of intelligence achieves omnipotence. The AI industry typically measures intelligence through benchmarks and model characteristics. Existing benchmarks become saturated over time, and new ones are continually released. New frontier models earn their status by making the benchmark numbers go up.

I take the LLM paradigm as given and use standard metrics of model capability and scale to illustrate my argument. As a result, this essay does not concern new types of AI models, including those that involve continuous or active learning, world models, or robotics.

I use three proxies for model capability and scale because no single metric is perfect.

• The number of model parameters: The most objective of the three metrics, because it concerns the properties of the neural network rather than its performance. For closed-source models, I use estimates of model size from the literature.

• The Artificial Analysis Intelligence Index: An average of many benchmark scores. It is a simple measure but risks becoming saturated.

• The Epoch Capabilities Index: An index that uses item-response theory to rank models. Importantly, it accounts for the fact that some benchmarks become saturated while others do not.

A note on scope: These measures capture raw model capabilities rather than the performance of products built around them. I therefore do not count scaffolding innovations such as Claude Code as model intelligence. Scaffolds can work with a variety of underlying models and are conventionally excluded when benchmarking frontier models.

Diminishing returns to intelligence

The simple intuition

Introductory economics students are often taught about diminishing marginal utility through the example of pizza. The first slice is amazing, the second is good, the third is fine, and the tenth is unpleasant. A similar phenomenon appears in business settings. Consider data science: improving an inventory forecasting model from 60% to 90% accuracy may be much more valuable than eking out the last few percentage points through extensive specialized modeling and PhD-level expertise. Diminishing marginal utility appears in many economic settings, and there is little reason to assume that model capability is an exception.

Why might measured model capability exhibit diminishing marginal returns? Once a model is capable enough to complete a task, additional capability may add little value. A given use case—say, a consumer chatbot—requires a model to attempt tasks of varying difficulty. As capability increases, the model can complete more of them correctly. But once most relevant tasks can be completed, other attributes, such as reliability, speed, and personalization, become more important. Benchmark saturation illustrates this threshold: beyond a certain capability level, nearly all leading models score close to perfectly. The empirical question I investigate is how much additional end-user utility comes from raw model capability rather than from these other attributes or from scaffolding.

Evidence from Arena AI

The ideal way to measure diminishing utility from model capability is to study people’s choices between outputs from models at different capability levels. Although comparable data from AI labs are not publicly available, Arena AI (formerly LMArena) collects users’ stated preferences over model responses and publishes a leaderboard. A user is shown two responses and votes for the one they prefer. These votes are used to calculate a publicly available Elo score. My analysis of Arena AI scores is illustrative, and I discuss its limitations at the end of the post.

Below, I plot the Arena AI score against the three proxies for model capability and scale: log parameters, the Artificial Analysis Intelligence Index, and the Epoch Capabilities Index (ECI).

Three observations stand out:

1. Greater measured capability is associated with higher user preference. The Arena AI Elo score increases with all three metrics.

2. The improvement associated with greater measured capability is gradual. A 10-point increase in the Artificial Analysis Intelligence Index is associated with an average increase of only 7 percentage points in the win probability implied by Elo. Users therefore still prefer the lower-scoring model’s response a substantial share of the time.

3. The relationship appears to exhibit diminishing returns, although the evidence varies by measure. In Panel A, log parameters are approximately linearly related to Elo. This implies diminishing returns to model scale: achieving the same increase in user preference requires roughly a tenfold increase in parameters. Panel B shows a similar flattening in the relationship between the Artificial Analysis Intelligence Index and Elo. The gains are large when moving from older Llama models (around 5 on the index) to GPT-4-class models (around 10), but the slope is modest between index values of 10 and 45. The most recent models—including Fable 5 and Sol—barely improve on the Elo scores of older, lower-scoring models such as Opus 4.6 and Grok 4.20. Panel C, which uses the ECI, provides weaker evidence of diminishing returns, although several current frontier models, including Fable 5 and GPT 5.6 Sol, fall below the fitted linear relationship.

Arena AI also breaks out its leaderboard by task category. The next figure shows how the preference advantage associated with greater capability varies across prompt categories. Each cell reports the average increase in win probability when moving from one capability bucket to the next. For example, the upper-left cell indicates that models with 10–100 billion parameters have a 9.2-percentage-point higher win rate, on average, than models with 1–10 billion parameters. Reading from left to right within each panel shows how this relationship changes at higher levels of the corresponding metric.

The shape of the returns to measured capability differs across categories. Math, for example, appears to exhibit relatively high returns compared with most other categories. By contrast, and perhaps surprisingly, coding queries appear to benefit relatively little from moving from the second-highest to the highest capability bucket. This heterogeneity suggests that diminishing returns may emerge at different capability levels for different use cases.

Reasons frontier models may remain valuable even with diminishing returns

Recursive self-Improvement

If end users experience diminishing marginal utility from additional intelligence within existing applications, frontier model developers must continually discover new and increasingly valuable uses for it. The most widely discussed (and meta) example of such a use case is recursive self-improvement (RSI), in which AI systems contribute to improving their own capabilities. If this process unfolds rapidly, it could generate gains in intelligence large enough to overwhelm ordinary diminishing returns. Even then, however, RSI would need to produce a persistently faster rate of improvement at the frontier than among models and firms attempting to catch up. A permanent speed advantage seems unlikely, given the difficulty of keeping secrets and the fact that even intelligent AI researchers face diminishing returns to research effort.

High-value problems and test-time compute

A second set of use cases involves exceptionally difficult problems and massive amounts of test-time compute. Noam Brown has emphasized that we have not yet seriously tried spending millions of dollars on a single inference, allowing a frontier model to reason for an extended period about one especially important problem. It remains unclear which problems would justify such expenditures, but leading candidates lie in areas such as biology and energy. An AI system capable of producing patentable cancer treatments or radically more efficient energy technologies could create enormous economic value. Model developers might capture that value through much higher inference prices, through licensing or intellectual property, or by vertically integrating into the industries in which the discoveries are applied. Anthropic’s recent acquisition of Coefficient Bio is suggestive of movement in this direction.

Zero-sum competition

Another set of use cases in which the returns to intelligence may not diminish sharply is when the user is engaged in a zero-sum competition. Whenever the best reasoner captures the prize, as in chess, financial trading, litigation, cybersecurity, or military strategy, what matters is winning and in many cases the best intelligence wins disproportionately. A small capability advantage may therefore remain valuable even when all participants are already capable. In such settings, demand for frontier intelligence may resemble an arms race: each participant must continue purchasing better reasoning simply to avoid falling behind.

The simplicity of one strong model and product

A final reason to use an expensive frontier model may, ironically, be cognitive overhead. Suppose a user performs a mix of tasks, some difficult and others relatively easy. The user may know that cheaper, less capable models could handle many of the easier tasks equally well, but may not want to spend time deciding which model to use for each request. Provided that the frontier model is not prohibitively expensive and comes packaged with a good interface, the user may therefore prefer to use it for everything, effectively paying a premium for convenience and consistency.

The size of the premium from such a bundle, however, is limited by the availability of increasingly capable routing technologies. If a routing system can identify the difficulty of each task and assign it to the least expensive model capable of completing it well, users can obtain much of the convenience of relying on a single frontier model without paying frontier prices for every query.

Even sophisticated routers are unlikely to eliminate the frontier premium entirely. Routing decisions are imperfect and users may therefore rationally favor systems that err toward more capable models, particularly when reliability matters a lot. In this case, frontier models may retain value as a form of task difficulty insurance.

Limitations of this analysis

This analysis is meant to be illustrative and to encourage AI labs and other companies to publish more work measuring these curves. Below are some reasons to be skeptical of my analysis.

Representativeness. Arena AI users are unlikely to be representative of the broader population of AI users, and the data comes from single-turn rather than multi-turn conversations. Business use cases may also exhibit different patterns from those observed here.

Causality. The analysis is descriptive and does not establish that greater model capability causes higher Arena scores. Models with different measured intelligence may also differ in response speed, style, or default reasoning budgets. If these characteristics affect user preferences and are correlated with measured capability, the observed relationship cannot be interpreted as the causal effect of capability alone.

Test-time compute. Arena Elo scores and intelligence metrics do not hold inference-time compute fixed. Many models have “thinking” or “reasoning” variants that spend more tokens (and dollars) before answering, and those settings often improve both benchmark scores and Arena win rates. A flatter Elo-versus-intelligence curve could therefore partly reflect differences in how much test-time compute each model was allowed, rather than diminishing returns to underlying capability alone. Benchmarks and Arena battles may also evaluate models under different default reasoning budgets, so the x- and y-axes are not always measured under the same compute regime.

Imperfect model matching. Linking Arena entries to Artificial Analysis, Epoch, and parameter estimates requires matching model names across sources that use inconsistent naming conventions. Matching can be imperfect for several reasons: the same weights may appear under slightly different labels (preview vs. release, date-stamped Arena IDs vs. catalog names); thinking and non-thinking configurations are sometimes scored separately on Arena but share a pooled model-group intelligence score elsewhere; closed-source parameter counts are estimates rather than confirmed figures; and some aliases are judgment calls when no exact identifier exists. When a match is wrong or only approximate, a point on the scatterplot pairs the wrong Elo with the wrong intelligence or size measure, which adds noise and can distort the apparent slope.

Comments welcome! Thanks to Tom Cunningham, Alex Imas, Abhishek Nagaraj, and Soumitra Shukla for early feedback.

Please cite:

@misc{fradkin_diminishing_returns,
  author       = {Fradkin, Andrey},
  title        = {Are There Diminishing Returns to AI Intelligence?}},
  howpublished = {Substack},
  year         = {2026},
  url          = {https://empiricrafting.substack.com/post/207306437}
}