When two tools disagree, which one is wrong?

Running two AI visibility tools in parallel is the fastest way to lose confidence in both. The numbers will differ, sometimes substantially, and the instinct is to conclude one is broken. Usually neither is.

The four sources of legitimate disagreement

Different prompts. The largest factor by far. Unless both tools use your identical prompt list, they are answering different questions.

Different sampling. A tool asking daily and averaging reports something different from one asking weekly, even with identical prompts.

Different mention definitions. Linked citation, unlinked brand mention, or both. This alone can double or halve a score.

Different model coverage. A blend weighted toward one assistant will diverge from a blend weighted toward another, and neither weighting is more correct.

A worked comparison

Imagine two tools report your ChatGPT visibility as 22% and 41%. On investigation: Tool A asks 40 generic category prompts once a week and counts only linked citations. Tool B asks your 20 prompts three times a week and counts any mention of your name. Both numbers are correct for what they measure.

Give Tool A your 20 prompts and ask it to count name mentions, and the gap will usually narrow sharply. What remains is sampling and timing, which is why you compare direction over several weeks rather than a single day's numbers.

How to resolve it

• Give both tools your identical prompt list. Much of the gap usually closes immediately.
• Ask each vendor for their mention definition in writing.
• Compare like assistants rather than blended scores.
• Compare direction over several periods rather than absolute values at a point.
• If a gap persists after all that, ask both for raw responses on the same prompt and date. The one that cannot produce them is the one to doubt.

What to do operationally

Pick one tool and stay with it. Consistency of methodology matters more than choosing correctly, because what you are managing is a trend rather than an absolute position.

Switching tools resets your baseline, which is the single most expensive thing you can do to a measurement program. If you must switch, run both in parallel for a quarter so you can stitch the series together honestly.

Using first-party data as a tiebreaker

When tools disagree about direction, look for evidence outside both. Bing Webmaster Tools' AI Performance report gives first-party citation counts for Microsoft's AI answers. Analytics shows whether referral sessions from assistant domains are rising or falling. Intake notes show whether customers mention assistants more or less often.

None of these measures exactly what the tools measure, but if they all point the same way as one tool and against the other, that is useful evidence about which tool's methodology fits your situation.

Documenting your method

Write down how your numbers are produced: which tool, which prompts, how many asks, which assistants, how a mention is defined, and when the method last changed. Keep it with the reports.

It seems like overhead until someone new joins, a vendor changes its defaults, or leadership asks why a number moved. Then it is the only thing that lets you answer with confidence rather than guesswork.

When to trust a single result

Sometimes one observation is enough. If an assistant states something factually wrong about your business, such as a closed location or a service you never offered, that is worth acting on after one sighting, because the source of the error is probably still out there.

Measurement questions (are we named more often?) need samples. Accuracy questions (is what it says true?) need only one clear example. Keep the two separate in your process.

Frequently asked questions

Why do AI visibility tools show different numbers?

Different prompt sets, sampling frequencies, definitions of what counts as a mention, and model coverage weightings. All four produce legitimate differences from the same underlying reality.

Which AI search tool should I believe?

The one you have used consistently. Absolute values are less meaningful than trend, so methodology consistency matters more than choosing the theoretically best tool.

Should I switch AI search tools?

Only with good reason, and run both in parallel for a quarter if you do. Switching resets your baseline, which is the most expensive thing you can do to a measurement program.

Can AI visibility data ever be exact?

No. Assistants vary their answers between sessions, so any figure is an estimate from a sample. Larger samples and consistent methods make estimates more reliable, not exact.

What should I do when a tool's numbers suddenly jump?

Check for a methodology change by the vendor, a model or product change by the assistant, and a change in your prompt set before assuming your work caused it.