Harvard Study: Claude Fable 5 Upgrade Barely Moved Developer Output
A Harvard study of 9,205 developers found Claude Fable 5 lifted merge rates by only 1.4 points versus Opus 4.8, despite an 11-point benchmark lead.
By Amara Osei
4 min read

What's News
- Fable 5 raised the PR merge rate by only 1.4 percentage points, statistically insignificant, versus Opus 4.8.
- The study tracked 297 Fable 5 users against 8,908 Opus 4.8 developers during the 19-day June White House ban.
- Five 2026 Claude upgrades covering 1,208 developers and 15,000+ code changes showed no significant productivity gain.
- Fable 5 scored nearly 11 points higher than Opus 4.8 on SWE-Bench Pro.
- Access to Fable 5 was restored on July 1 after a 19-day suspension.
By Amara Osei
Access to Anthropic's higher-scoring Claude Fable 5 raised developers' code merge rate by just 1.4 percentage points, a Harvard University study has found — a difference the researchers call statistically insignificant and practically a rounding error. Fable 5 had scored nearly 11 points higher than its predecessor Opus 4.8 on the SWE-Bench Pro coding benchmark.
The study, titled Higher-Scoring Coding Models Did Not Get More Work Done, exploited an unusual natural experiment: the 19-day White House ban on Claude Fable in June. Developers submitting pull requests to GitHub downgraded to less powerful Claude versions. Because all Claude versions co-sign pull requests via the "Generated with Claude Code" notice, the GitHub API exposed who used which model, and when.
What did the ban actually change?
Almost nothing, according to the data. The researchers tracked 297 Fable 5 users before, during and after the ban, and compared them with 8,908 developers who kept using the older Opus 4.8 model. The main productivity metric was the proportion of submitted code changes accepted into GitHub projects within seven days.
The timeline split into four periods: before Fable 5's June 9 release; its three days of availability; the 19-day suspension; and the restoration of access on July 1. The control group allowed the researchers to separate the ban's effects from wider fluctuations in GitHub activity.
The paper states: "Results show that having access to Fable 5 changed the merge rate by only 1.4 percentage points, and the change was not statistically [significant]. We also analyzed five of the eight model upgrades in 2026 and found that the merge rate did not rise after these upgrades [either]."
It adds: "Similarly, developers did not get more pull requests merged per week. These results show that gains on current coding benchmark did not really translate into more finished work."
How big was the wider sample?
The broader analysis covered five Claude model upgrades during 2026, involving 1,208 developers and more than 15,000 proposed code changes. No statistically significant improvement appeared in merge rates or in the number of changes completed each week. The merge rate actually fell slightly, by an estimated 0.9 percentage points.
Developers on newer models completed just 2.4% more accepted submissions per week — too small to establish a reliable gain. Even where benchmark scores had risen by between 5 and 18 points, the study found no significant productivity improvement. Failure rates in GitHub's automated code checks stayed essentially unchanged.
The dataset spanned November 2025 to October 2026, with at least four weeks allowed for the latest submissions to be merged. Additional checks covered PRs merged within an hour, PRs closed without merging, comment counts, changes subsequently reversed, and automated-check failures.
Why don't benchmark gains show up at work?
The paper's central explanation is simple: the tasks developers assign to AI agents already fall within older models' capabilities. "We hypothesize that the main reason Fable 5's gains on benchmarks did not translate to real-work productivity is that the tasks developers give agents are already within what the older model can do," the authors write.
They also flag methodological traps. Early adopters of new models may simply be more experienced, making it impossible to separate the model from the person. And developers may reserve the strongest model for the hardest tasks — which take longer and are less likely to merge, making the strongest model look less productive on almost any measure. Earlier studies, the paper contends, were skewed because most code changes came from a small number of projects, each with its own acceptance rules.
The warning extends beyond developers. "If regulators and buyers judge how much stronger a model is by its benchmark [score], the difference they rely on will likely not show up in user productivity," the study concludes.
The findings land in a sector the source article describes as attracting huge global interest while operating under maximum corporate opacity. Benchmark contamination compounds the problem: test data tends to leak into the training data of the next model, breaking the template for consistent measurement. The persistence of long-outmoded standards such as ImageNet in computer vision research illustrates how slowly the field replaces broken metrics.
For buyers weighing AI subscriptions, the takeaway is concrete: a double-digit benchmark lead did not produce measurable output gains across more than 15,000 real code changes. Until vendors publish work-based evidence, the score gap may not be worth the upgrade price.
Source: unite.ai
More from Amara Osei
Show full bio
Senior reporter covering companies, deals and strategy in entrepreneurship, startups and venture funding at Business Bearings.
2,274 articles