Innovation & Tech

When ChatGPT, Gemini and Perplexity Agree, They Can Still Be Wrong

Scento's 300-query study found ChatGPT, Gemini and Perplexity agreed on identical wrong perfume pyramids — consensus measures spread, not accuracy.

By Amara Osei

5 min read

When AI Engines Agree, That Is Not Proof: What 300 Product-Fact Queries Show
When AI Engines Agree, That Is Not Proof: What 300 Product-Fact Queries Showpixel2013 / pixabay

What's News

  • Scento's 20 October 2026 study ran 300 product-fact queries across ChatGPT, Gemini and Perplexity covering 100 perfumes from 45 houses.
  • All three engines returned every official note for 59 of 99 perfumes but matched the official list exactly for only 39.
  • For Jean Paul Gaultier Le Male Elixir, all three engines produced an identical seven-note pyramid against a three-note reference.
  • Gemini added unique off-list notes in 32% of answers, versus 10% for ChatGPT and 11% for Perplexity.
  • Designer perfumes drew off-list notes 22 to 32 percentage points more often than niche ones across all engines.

Three major AI engines returned every official perfume note for 59 of 99 fragrances, yet matched the official list exactly for only 39. That gap sits at the center of Scento's October 2026 study of how ChatGPT, Gemini and Perplexity answer product-fact questions — and it undercuts a popular safety net in AI agent design.

The common tactic: ask more than one model, or the same model several times, and trust the answer they agree on. The idea has roots in real research. Self-consistency, which samples multiple reasoning paths and keeps the majority answer, improves accuracy on reasoning benchmarks. Using one model to grade another, studied in the "Judging LLM-as-a-Judge" paper, reaches high agreement with human raters — though its authors also document position, verbosity and self-enhancement biases.

Scento tested whether agreement means accuracy on product facts, where the right answer is usually published by the maker. The verdict: only partly.

How the Test Worked

The prompt was a single sentence asking for a fragrance's top, heart and base notes. Scento sent it for each of 100 perfumes, drawn from 45 houses, to the three engines on 20 October 2026: 300 calls in total. All ran once, in English, from a US location. Gemini returned 99 answers; one call ended in a service error.

For 91 perfumes, the reference was the note list the house publishes on its own site. For the other 9, it was a list two authorised retailers agree on. A script using no language model at all did the grading. It mapped every note onto a 411-note glossary and compared the result with the reference. A manual check of 20 gradings found 17 exact agreements and 19 within one note — though the check wasn't blind.

The study measured recall, precision, exact match and placement — whether each note sits in the right top, heart or base tier.

What Does Engine Agreement Actually Prove?

On recall, the engines look almost interchangeable. Median recall was 1.00 for each. ChatGPT returned every official note in 82% of answers, Perplexity in 78% and Gemini in 69%.

But full consensus was much rarer, and the shared answers were the instructive ones. For Jean Paul Gaultier Le Male Elixir, the two agreeing retailers give three notes: lavender on top, tonka bean in the heart, benzoin in the base. All three engines produced an identical seven-note pyramid. Each included all three reference notes, put tonka bean and benzoin in each other's tiers, and added four more.

For Viktor&Rolf Flowerbomb, the house lists four notes. Gemini and Perplexity both returned the same 11-note pyramid, missed one official note and added eight. Perplexity's answer even noted that the brand's current description highlights the four official notes — so it held both versions and led with the older one.

The data can't say why the answers match. The practical point doesn't depend on the cause. "When systems draw on the same pool of web text, a majority vote measures how widely a version has spread, not whether it is current," the study concludes.

In roughly one case in five — 20 of the 99 perfumes — all three engines returned every official note, yet at least one also added off-list notes. A recall-only check would have passed all three.

Which Errors Does Cross-Checking Catch?

One-engine outliers are the easy ones. In 32% of its answers Gemini added at least one note neither rival named, against 10% for ChatGPT and 11% for Perplexity.

For Prada Infusion de Vétiver, the house lists four notes. ChatGPT and Perplexity returned exactly those four, in the right places. Gemini returned twelve, including eight no other engine gave, for a precision of 0.33. For Rochas Eau de Rochas, Perplexity matched the ten-note list exactly while Gemini returned 18 notes, ten of them unique.

The asymmetry, per the study: "Disagreement is a useful alarm. Agreement is not a certificate."

Structure is its own failure mode. For Montblanc Explorer, ChatGPT matched six of nine official notes, but only two sat in the correct tier — a placement accuracy of 33%. Any evaluation that only asks whether a fact appears somewhere in the answer will miss this class of mistake.

Where Errors Cluster

Designer perfumes drew off-list notes more often than niche ones on all three engines, with gaps of 22 to 32 percentage points. ChatGPT showed the widest split: four in ten of its designer answers included at least one off-list note, versus fewer than one in ten for niche.

Controlling for list length, the pattern held: 70% versus 31% extras for lists of five notes or fewer, 61% versus 21% for six to eight, and 26% versus 13% for nine or more. The samples are small, and each group drew from its own set of houses, so the cause isn't isolated. The takeaway: accuracy varies by product segment, and a single catalogue-wide figure can hide that.

What Should Shopping-Agent Builders Do?

Scento's recommendations are concrete:

  • Use consensus to find disagreement, not to confirm truth.
  • Ground owned facts in an owned source — structured records carrying their source page, via schema.org Product markup and the Model Context Protocol. Google's guidance says structured data should match visible page text, and no special markup is needed to appear in AI features.
  • Score precision, exact match and structure, not just recall.
  • Grade deterministically where reference data exists; a rule-based grader is repeatable, auditable and doesn't share a model's blind spots.
  • Run more than once, since answers vary run to run — a point Andrew Lissimore made on this site about measuring AI shopping recommendations.

The limits are real: one call per perfume per engine, one day, one language, one country. Claude and engines without web search weren't tested. A note missing from an official list isn't automatically wrong, since houses sometimes publish only key notes, so precision is a lower bound on correctness.

Within those limits, the pattern held. The study's closing question for anyone whose agent checks itself against a second model: "What does your agent do when two models give the same wrong answer?"

Source: unite.ai

Share this article:

More from Amara Osei

Amara Osei

Show full bio

Senior reporter covering companies, deals and strategy in entrepreneurship, startups and venture funding at Business Bearings.

2,267 articles

Related articles

« Previous articleNext article »