Why one AI answer proves nothing

Someone screenshots ChatGPT recommending a competitor and treats it as a finding. Someone else screenshots it recommending them and puts it on a slide. Both screenshots are close to worthless, and the reason is measurable.

The core finding

Response length varied too, from two or three items to more than ten, with no discernible pattern. The assistants are not returning a ranking that occasionally wobbles. They are sampling from a distribution every single time.

Where the noise actually comes from

A 2026 variance-components study decomposed the non-determinism in brand answers. The result is uncomfortable for everyone selling in this category, including us.

The same study produced a finding that changes how sampling should be designed: adding diversity of wording and model reduces relative-error variance roughly fifteen times more than five additional repeats of the same prompt. Asking a different question is worth far more than asking the same one again.

It also sets a ceiling worth knowing. A design spanning eight languages, three models and fifteen paraphrases reaches a reliability of about 0.36. Better, and still modest. Nobody in this field is measuring precisely.

What this means for the tools

Fishkin's operational guidance is to run each prompt 60 to 100 times or more, implying roughly 20 runs across 7 phrasings and 7 models for accuracy within a percentage point. Set that against commercial tools offering 25-prompt or 15-prompt tiers.

Most AI visibility products are sampling one to two orders of magnitude below what their own headline precision implies. That does not make them useless. It makes their decimal places fictional.

Three traps that survive better sampling

Whoever picks the prompts picks the result

The same brand, on the same underlying data, can be shown at 20%, 16.8% or 31.4% share of voice depending on which prompts were chosen and which scoring formula was applied. A provider that selects its own prompt set and then reports improvement against that set has a plain conflict of interest. Ask to see the prompts, and ask whether they changed between reports.

The API is not the product

Answers pulled through an API reflect one account, one geography, one subscription tier and one moment of one model's state. Real users see a distribution shaped by personalisation, location, account state and constant product changes. Anyone measuring this way, ourselves included, is using a proxy and should say so.

Model drift looks exactly like your work

Providers ship model and retrieval updates constantly. One documented case saw GPT-4's accuracy on a fixed task fall from 84% to 51% in three months. A client's visibility can move materially in either direction for reasons entirely unrelated to anything an agency did — and it is genuinely difficult to tell the two apart.

The counter-argument, and what it does not answer

Agencies in this category have a standard response to the reproducibility problem, and it is worth engaging with properly because part of it is correct.

The argument runs: the criticism of tracked prompts is that they are invented questions rather than real conversations, and that objection is weakening. Prompt sets are increasingly built from Google Search Console queries, GA4 landing pages and on-site search terms, and some vendors now license anonymised real prompts from opt-in consumer panels. Scale is not a constraint either — practitioners track hundreds of prompts, not fifteen.

All of that is true, and a prompt set grounded in real buyer language is genuinely better than one somebody made up. But it answers a different objection to the one the research raises.

The variance decomposition is what makes this stubborn. Re-running an identical prompt accounts for 34.8% of the variance on its own. Wording accounts for another 32%. Neither of those is addressed by sourcing the prompt more carefully — a perfectly sourced prompt still lands in a distribution.

Scale helps, and less than people assume. The same study found that a design spanning eight languages, three models and fifteen paraphrases reaches a reliability of about 0.36. That is a large, expensive panel, and it is still a long way from the precision implied by a dashboard reporting share of voice to one decimal place.

So the honest position sits between the two. Tracked prompts are not guesswork, and the people saying so are right. They are also not precise, and a report that gives you a number without a sample size and an acknowledgement of variance is overstating what it knows — whatever the prompts were built from.

So which numbers survive?

The line between a defensible measurement and theatre is sharper than the category admits.

How this shapes what we do

Two things follow directly, and both are visible in how our own reports are built.

1. We spread runs across different wordings and different locations rather than repeating one question, because the variance research says diversity buys far more reliability than repetition does.
2. We report counts out of a stated denominator and never a score out of 100. A count you can go and check yourself is more use than a composite number we invented, and given a single-answer reliability of 0.01, a score would be a decimal place laid over a coin flip.

It also sets the honest limit on what any report of this kind is. It is a record of what we observed on the dates stated, not a forecast, and running it again later will produce different figures. That is a property of the systems, not a defect in the method.