Why Did the AI Improvement Rate Drop from 74% to 56%?

A headline says AI improvement fell from 74% to 56%. That sounds like a clear slowdown. It may be—but only if both percentages measure the same thing, on the same task mix, against the same baseline, using comparable releases.

That is a demanding condition. “Improvement rate” is not one standard industry metric. It might mean the share of new models that beat their predecessors, the percentage of blind comparisons won by a model, or the proportion of launches that clear a benchmark threshold. Those measures answer different questions.

So the useful question is not simply whether AI is improving more slowly. It is: what changed in daily use, and how was that change measured? This matters to teams deciding whether to upgrade, especially when a new release costs more, behaves differently, or regresses on work they rely on.

First, define what “74% to 56%” means

Before interpreting the figures, identify the numerator and denominator. If 74% means “74% of evaluated releases beat their predecessor,” while 56% means “56% of blind-vote comparisons preferred the new model,” the apparent drop is not a clean trend. One measures releases; the other measures preferences.

Even when the label stays the same, the underlying comparison can change. A result may depend on which models were included, how ties were counted, whether the tests covered text or multimodal tasks, and whether evaluators saw model names. A percentage without that method is a headline, not yet evidence of a trend.

Possible meaning of “improvement rate” What it tells you What it does not tell you by itself Share of releases that beat a prior model How often evaluated launches move a chosen score upward How large the gains are, or whether users notice them Blind-vote win rate Which answer people prefer in a particular comparison setup Whether the preferred answer is more accurate, cheaper, or safer Benchmark score change Performance on a defined set of tasks How well the score transfers to a company’s own workload

Preference is not the same as capability

LMArena, formerly LMSYS Chatbot Arena, uses anonymous side-by-side comparisons in which people vote for the response they prefer. Its text leaderboard with style control is useful because response style can affect votes. A model that is more fluent, concise, or engaging may win more often even if it is not more reliable on factual or technical tasks.

That does not make blind-vote results unhelpful. They provide evidence about human preference under the arena’s conditions. But preference is not a universal proxy for capability. Voters may reward tone, formatting, and a satisfying answer. They may not verify a citation, test code, or check whether a long answer contains a subtle error.

The Hugging Face dataset lmarena-ai/leaderboard-dataset gives analysts a way to inspect leaderboard-related data and reproduce parts of an analysis. It is a useful starting point, not an automatic answer. Anyone calculating an “improvement rate” from it should document the dataset version, model inclusion rules, date window, tie handling, and comparison method.

Announced dates and usable dates are different

AI release timelines often mix several dates: an announcement, the start of a limited rollout, general availability, and the date a particular user or API account could actually access the model. Treating those as interchangeable can distort a year-over-year comparison.

For example, an announced model may not yet be available to most users. A model that appears in a leaderboard may have been tested before broad access. And a release can reach different products or regions on different schedules. If the analysis counts every announcement as a measured release, it may include models that have not received enough real-world or independent evaluation.

A practical release record should keep separate fields for:

Announced date: when the company publicly described the model.
First verified access date: when an independent user could confirm access through a stated product or API.
General availability date: when access was broadly offered, if the provider publishes one.
Measurement date: when the relevant benchmark or preference data was collected.

This is the sort of distinction that makes release tracking useful rather than decorative. A measurement tied to a verified access date is easier to compare with another measurement taken under similar conditions. An announcement date alone says little about when users could try the model.

Why the measured improvement rate could fall

1. The easy gains may already be behind the latest releases

When a model family is far from a chosen benchmark ceiling, a new training run may produce a visible score jump. As performance gets stronger on that benchmark, each additional point can become harder to earn. The improvement rate can fall even while models continue to improve.

That is one possible explanation—not proof that progress has slowed across every task. A benchmark can saturate while performance on long-context work, tool use, or specialized workflows changes in a different direction.

2. More releases create more chances to record a miss

Release cadence changes the denominator. If OpenAI, Anthropic, and other labs ship more versions, snapshots, or product variants, an analysis may count more measured releases in a given period. Exactly.. Some may be targeted improvements rather than broad upgrades. If each is judged against one predecessor on one aggregate score, the share that wins can decrease even as the range of available capabilities grows.

Frequent releases also make “the predecessor” ambiguous. Is the comparison against the latest generally available model, the previous API snapshot, or the closest model by size and price? Without a consistent rule, improvement rates can reflect bookkeeping as much as technical progress.

3. Launch benchmarks and blind votes reward different things

Labs commonly publish launch evaluations chosen to test particular capabilities. Those evaluations can be valuable, but a lab’s selected tests are not the same as an independent blind-vote leaderboard. The task mix, scoring rules, prompts, and comparison models may differ.

For a credible claim, keep the two evidence types separate. A launch benchmark can show that a model improved on specified tests. A blind-vote preference result can show that voters preferred its responses in a defined comparison. Neither should be presented as a direct substitute for the other.

4. The upgrade may help one workflow and hurt another

Model updates can bring regressions alongside gains. I've seen this play out countless times: thought they could save money but ended up paying more.. A new release may reason better on one task but follow formatting instructions less consistently, change how it handles refusals, or produce code that needs more review. An aggregate score can hide those trade-offs.

That is why regression rate 2026 should be measured alongside improvement rate 2026. Teams should define regressions in terms of tasks that matter to them: a higher error rate, a lost integration, extra editing time, increased latency, or a change in refusal behavior. A model can win on average and still be a worse upgrade for a specific workflow.

Price belongs in the upgrade decision

A small quality gain may not justify a higher bill. One reported comparison says GPT-5.2 costs about 40% more than GPT-5.1. Treat that as a lab claim reported by a secondary source, not as a verified, universal price fact. Before using the figure in a procurement decision, confirm the source, the pricing date, and whether the comparison concerns input tokens, output tokens, or a particular service tier.

Pricing can vary by product, usage tier, and date. The right comparison is the cost of completing the same workload to the same quality level—not simply the per-token rate. If the newer model needs fewer retries or less human review, it could still be cheaper in practice. If it upcoming AI model releases adds cost without improving a team’s results, a higher leaderboard position may not matter.

How to measure releases without confusing the result

Ask yourself this: teams tracking measured releases can make the analysis more useful with a small, consistent protocol:

Choose one question. Decide whether the study concerns user preference, task accuracy, cost, speed, or release-level wins. Do not merge them into one percentage.
Freeze the comparison rule. Compare each new model with a named baseline, such as the previous generally available version, and state how ties are handled.
Record dates separately. Log announcements, verified access, general availability, and measurement dates rather than collapsing them into a single release date.
Use a representative task set. Include the work users actually do. Keep prompt versions and scoring rules stable across model versions.
Track regressions by task. Report where the new model loses, not just its average score. A category-level result is often more actionable than a single composite number.
Include cost and latency. Measure the full workload, including retries, human review, and time to completion.
Repeat the test. Small changes can be noise. Report sample sizes and uncertainty, and avoid treating a narrow lead as a dependable win.

So, did AI progress slow down?

A drop from 74% to 56% could indicate that fewer measured releases beat their predecessors under a particular method. It does not, on its own, prove that AI capability is improving more slowly overall. The change could reflect a tougher comparison set, a different release cadence, benchmark saturation, mismatched dates, or a shift from launch evaluations to blind-vote preferences.

The practical conclusion is more modest and more useful: stop treating every new version as an automatic upgrade. Compare verified, usable releases against the model your team actually uses. Test for task-level gains and regressions. Include cost. Keep announcement dates separate from measurement dates.

That gives “improvement rate 2026” a meaning that can support a decision—and makes “regression rate 2026” visible before a version bump reaches production. The headline may be 74% to 56%. The upgrade decision should be based on what changed in daily use.