Innovation & Tech

GPT-4's Accuracy Fell From 84% to 51% — And Businesses Paid the Bill

GPT-4 accuracy fell from 84.0% to 51.1% in three months under the same API. Studies show silent updates break working predictions 87.9% of the time.

By Grace Kim

5 min read

Your Business Is Losing Money on AI You Don’t Control — Here’s Why.
Your Business Is Losing Money on AI You Don’t Control — Here’s Why.Lukas Blazek / unsplash

What's News

  • GPT-4 prime-number accuracy fell from 84.0% in March 2023 to 51.1% in June 2023 under the same model name, per Stanford and UC Berkeley research.
  • A 2024 Carnegie Mellon/HKUST study found 58.8% of prompt-model combinations lost accuracy after API updates; silent changes broke working predictions 87.9% of the time.
  • Anthropic faces class action No. 3:26-cv-07699-RFL over alleged degradation of Claude Pro, Max 5x and Max 20x tiers between March 4 and May 6, 2026.
  • A routing error grew from 0.8% to 16% of Sonnet 4 traffic, hitting roughly 30% of active Claude Code users, Anthropic acknowledged in September 2025.
  • OpenAI closed a round at an $852 billion valuation; Anthropic prepared for a listing above $2 trillion with roughly $895 billion in compute commitments.

GPT-4's prime-number accuracy fell from 84.0% in March 2023 to 51.1% in June 2023 under the same model name and API. Stanford and UC Berkeley researchers Lingjiao Chen, Matei Zaharia and James Zou documented that drop in Harvard Data Science Review after testing identical prompts on GPT-4 and GPT-3.5 across snapshots three months apart. Businesses renting AI through APIs absorbed the damage — and kept paying.

The chain-of-thought boost collapsed from 24.4% to -0.1% in the same window. On a happy-number task, GPT-4's accuracy fell from 83.6% to 35.2%. Directly executable code plummeted from 52.0% to 10.0% because extra formatting broke pipelines. Instruction-following on answer extraction collapsed from 99.5% to 0.5%.

Same model name. Same API. Very different product.

How often do silent updates break your workflows?

A 2024 study by Wanqin Ma, Chenyang Yang and Christian Kästner of Carnegie Mellon and HKUST quantified the pattern: 58.8% of prompt and model combinations lost accuracy after an API update. Of those drops, 70.2% exceeded 5 percentage points. Across updates, 10.9% of individual predictions regressed — and 63.8% of those failures occurred at maximum confidence.

That last number is the killer. The model was most sure of itself exactly when it was wrong. The study found silent changes broke working predictions 87.9% of the time, even in updates where overall accuracy rose.

Researchers Emanuele La Malfa and colleagues at Oxford and the Alan Turing Institute went further in a 2025 AAAI paper, first published in the Journal of Artificial Intelligence Research. They warned that language-model providers deprecate models on short schedules, collect prompts under commercial licenses, and alter serving layers while keeping your pinned model ID unchanged. The customer takes the hit and reinvests in adapting to a model that isn't smarter — just cheaper for the provider to run.

What is Anthropic accused of?

Anthropic is defending a consolidated consumer class action in California federal court, In re Claude Service Degradation Litigation, No. 3:26-cv-07699-RFL. The suit alleges the company wrongfully degraded its paid tiers — Claude Pro at $20/month, Max 5x at $100/month and Max 20x at $200/month — between March 4 and May 6, 2026.

In April 2026, the problem became visible enough that Anthropic published an anti-crisis investigation into Claude Code complaints. The company pointed to three changes that intentionally degraded output without touching the model itself:

  • Default reasoning effort lowered from high to medium
  • A bug that discarded earlier reasoning after idle sessions
  • A brevity instruction that hurt coding output

Earlier, in September 2025, Anthropic acknowledged three infrastructure bugs that corrupted responses and misrouted traffic. One routing error grew from 0.8% of requests to 16% of Sonnet 4 traffic and hit roughly 30% of active Claude Code users. Customers paid full price for degraded service; transparency arrived only after complaints.

Who profits when your tools break?

The financial incentives behind this opacity are enormous. OpenAI closed a funding round at an $852 billion valuation and discussed a round above $1.2 trillion. Anthropic prepared for a public listing above $2 trillion alongside roughly $895 billion in non-cancellable compute commitments.

High switching costs serve those valuations. Every broken workflow means more API calls to test, more compute for quality assurance, and more tokens to rewrite system prompts — so your business isn't sued by customers over a product change you don't control.

There's classic economics behind the trap. In 1970, Nobel laureate George Akerlof showed in The Quarterly Journal of Economics how quality uncertainty creates a "market for lemons." A 2026 CHI study by Alexander Erlei and colleagues at Göttingen, Cagliari, TU Delft and Columbia proved AI buyers fall into it. Without disclosure in a high-lemon market, 92.2% of delegations went to low-quality AI systems. Partial accuracy disclosure in a low-lemon market cut lemon delegations to 5.5% and raised earnings by 36.4 coins.

Will regulators step in?

Not soon. Serena Booth argued at ICML 2025 that consumer protection laws apply to AI. But a 2026 FAccT study by Abeba Birhane, William Agnew and colleagues found 249 instances across 100 news articles where AI firms used 27 distinct capture mechanisms to fight oversight. A 2026 report by Miles Brundage and 47 co-authors noted that frontier AI receives less independent scrutiny than consumer goods, with baseline AAL-1 audits costing $300,000 to $600,000 and AAL-2 reviews exceeding $1 million.

What can businesses actually do?

The author, drawing on nearly two decades in the field, argues tier-one superintelligence models run on a domestic hardware unit costing less than three months of tokens. For drafting, classification, summarization, contract review and code assistance, an open-weight model on local memory keeps prompts inside your building.

For companies staying on APIs, the article prescribes four defenses:

  • Have engineers hash weights, lock configurations, cut token bills and guarantee tools won't lose accuracy overnight
  • Build a ten-prompt regression baseline for revenue- or liability-tied workflows; log outputs, timestamps and error rates monthly across data slices
  • Audit switching and token costs over a ninety-day window; count hours spent fixing broken prompts and compare that bill against a local workstation
  • Demand contractual change notices and data sovereignty — if a provider won't guarantee advance notice before altering reasoning effort or routing, move proprietary workflows to weights you control

The backdrop: OpenAI's CEO suggested on the Cheeky Pint podcast that processing personal data on the company's data centers would be more efficient than on personal computers, then walked the remark back after backlash. The author's closer is blunt — copy the article into a vibecoding tool and it will implement most solutions within 20 minutes, because not addressing the issue today might cost a business everything tomorrow.

Original: arxiv.org

Share this article:

More from Grace Kim

Grace Kim

Show full bio

Market editor covering people, launches and day-to-day practice in entrepreneurship, startups and venture funding at Business Bearings.

2,178 articles

Related articles

« Previous articleNext article »