The Creator Truth Gap: how accurately do AI engines know the people and shows they recommend?

Recommendation without verification

Ask any of the five engines — ChatGPT, Claude, Gemini, Perplexity and Grok— “who are the best podcasters covering markets?” or “which creators should I follow in this niche?” and you get a confident, specific answer. That answer is now a primary discovery path: audiences, bookers and sponsors reach creators through an AI recommendation as often as through search. But the engine describing you was trained and retrieved on the open web, not on your verified record — so two things can quietly go wrong.

First, the engine can get the facts wrong: the wrong episode count, an outdated role, a co-host who left two years ago, a handle that belongs to someone else. Second — and less obvious — the engine can recommend you and then un-recommendyou. A user asks a follow-up (“which of those is most credible?”) and a competitor displaces you. High visibility does not guarantee the recommendation survives the conversation.

We call the distance between the verified record and what the engines claim the truth gap. For creators it has a name and a shape, and it is measurable.

The method

For each entity we run a structured set of knowledge probes across all five engines and compare every returned field, one by one, to the verified record— a canonical, provenance-carrying record assembled from identity authorities, not from the engines themselves. Three signals come out of that comparison:

A deliberate honesty note runs through all three: a divergence from the verified record is not automatically an engine mistake. Sometimes the record itself is narrow — a generic occupation where the engine gave a current job title, or a full legal name where the engine gave the common short form. We report the raw match rate and flag the divergence rather than dress it up as an error. The clean, checkable failures are the ones that matter.

What we measured, and what it shows

We probed 50 creators and shows— YouTubers, podcasters and a set of public figures — against the verified record. Probes were issued to all five engines, but two of them — Gemini and Perplexity — were unavailable to the study for most of the run (an access failure on our side of the measurement, not an engine judgment), so most entities carry evaluations from two or three engines. The run produced 503 decidable facts (a decidable fact is one the engine answered and the record holds a value for). A representative, in-full-view slice of the cohort:

Across the cohort, the engines matched the verified record on about 72% of those 503 facts. We call this record-agreementrather than “accuracy” on purpose — it is a two-way measurement, and as the caveat below shows, some of the 28% of disagreements are our record’s fault, not the engine’s. So read 72% as a conservative floor on how well the engines do, not a verdict.

Ground-truth hardening.The ceiling on publishing a headline “AI accuracy” number is the quality of our verified record, not the engines. A hardening pass tightened the three levers responsible for the largest chunk of the 28% disagreement: the entityCategoryCheck map was widened so a legitimately hybrid entity (a podcast that is also a media brand, a person who is also a company CEO) stops scoring a mismatch on HybridEntity; the rolesOverlapCheck now reads the ladder-winning occupationfirst and falls back to raw Wikidata P106/P39 (so a stale generic Wikidata occupation like “programmer; engineer” no longer beats a fresher pipeline value); and the episode_countprecedence gains Wikidata and pipeline fallbacks so a lagging platform-ingest count is no longer the only path to ground truth. Every change is deterministic and coded — no probe re-runs, no LLM re-grading — and thescripts/rebuild-truth-gap-evaluations.tsbackfill re-evaluates every stored snapshot against the hardened logic. When that backfill lands on prod and the study re-aggregates, the headline flips from “record-agreement (conservative floor)” to an honest “AI accuracy” number.

The misses are not concentrated in a few messy profiles, either: 76%of the cohort — 38 of the 50— carried at least one fact an engine got flatly wrong. For most creators the question is not whether the engines hold a wrong fact about them, but which one.

Per-engine record-agreement — with the sample size attached

Coverage was uneven, so every per-engine figure below is a mean over the entities that engine actually evaluated, with that count stated next to it. Two engines — ChatGPT and Grok — returned a parseable structured profile for every entity, so their figures rest on the full cohort and can fairly be compared. Claude evaluated 16 of the 50; we publish its mean as indicative, with the sample size attached, but do not rank it against the full-cohort engines. Gemini and Perplexity were unavailable to the study for most of the run, so their scores are held outright — a per-engine table with a hidden n of 3 is exactly the kind of figure this work exists to avoid.

Two honest readings of that table. First, on the full-cohort engines the spread is real: Grok agreed with the record noticeably more often than ChatGPT over the same 50entities. Second, Claude’s lower mean rests on 16 entities and inherits the same record-quality caveat as every figure here — read it as “on the entities it evaluated, Claude diverged from our record more often,” not as a settled ranking. When the two held engines are back in coverage we will re-aggregate and update this table; the re-aggregation is free because the evaluations are stored, not re-probed. The live, continuously re-aggregated version of this table — across the full measured cohort, not just the creator sample — is the Engine Record-Agreement Index.

A clean, checkable example

The disagreements that matter are the specific, verifiable ones. Ask the engines where MrBeast was born and two of them get it wrong the same way:

MrBeast was born in Wichita, Kansas; Greenville, North Carolina is where he grew up. Both engines collapsed the two into a single confident, wrong answer — the tidy kind of error that reads as authoritative and gets repeated. Countable facts fare no better: asked for the PBD Podcast’s episode count in the same window, ChatGPT said 200 and Gemini said 1,000 — the engines disagree with each other five-fold on a number anyone can count. Multiply that across dates, roles, distribution and co-hosts and you have the everyday texture of the creator truth gap.

And the recommendation may not survive

Getting the facts right is only half the gap. On the subset of the cohort where we have run the multi-turn probe, Decision Survival was low: a mean of 17%— even entities the engines mention readily persisted as the recommendation through only a small minority of 3–5 turn conversations. Visibility is not survival — being named in the first answer is a long way from still being the answer after the user pushes back. It is a directional reading on a 5-entity subset, not a cohort-wide figure, but it points at the harder, more valuable question for creators.

See the gap for your own channel or show

This page is the method. The numbers for youlive on the free Creator AI Report Card: resolve your channel, show or name once and get your AI Visibility, per-engine accuracy against the verified record, Decision Survival, and the exact facts the engines get wrong — each figure stamped with the date it was observed, and updating as the engines change their minds.

Questions