The GEO Directory · 18 November 2026 · Published

Your AI visibility score went up. Did anything improve?

Before you buy more tracking or celebrate a higher score, check whether you measured the same thing twice.

THE DENOMINATOR TRAP Illustrative data / not a product screen

Same four mentions.
A much bigger percentage.

Period A40%

10 of 12 complete
4 observed mentions

Period B67%

6 of 12 complete
4 observed mentions

Mention No mention Missing / unknown
The score rises because fewer answers return. Not because another answer names the brand.

A founder opens a weekly report. AI visibility has risen from 40% to 67%. The tempting next move is to credit the new content, renew the tool and do more of what worked.

But four mentions out of ten completed answers is 40%. Four out of six is about 67%. A dashboard can rise without recording one additional mention.

This does not make AI visibility tracking useless. It makes the question behind the percentage important: which answers, collected under which conditions, are in this number? A score is useful when it supports a decision. It is not useful when a collection problem looks like a marketing win.

01 / Define the outcome

Four signals, four different jobs

Start by naming what you want to improve. These signals can move in different directions, and a single blended score hides why.

A practical measurement vocabulary, not a universal vendor standard

Brand mention

What you observed
The answer names your brand or a verified alias.
What it does not prove
That the answer recommends you. A warning also mentions you.
Useful next action
Read how the brand is described.

Recommendation

What you observed
The answer presents your product as a suitable option for the stated need.
What it does not prove
That your site is linked, or a buyer followed the advice.
Useful next action
Check fit, qualifications and factual accuracy.

Link to your site

What you observed
A returned source or citation links to a URL on your domain.
What it does not prove
That the answer recommends your product. Your article might support a point about a competitor.
Useful next action
Inspect which page and claim the link supports.

Referral visit

What you observed
Analytics records a visit attributed to an AI source.
What it does not prove
That you know the exact prompt, every prior touchpoint, or incremental revenue.
Useful next action
Check landing pages and downstream actions.

OpenAI documents that web-search responses may contain citations, and cautions that sources can be incomplete or wrong. A link needs checking; it is not an endorsement certificate. [2]

A report about unaided discovery also needs to keep branded questions separate. Asking an assistant whether your product is good gives it your name. That can be a useful reputation check, but it is not the same test as asking which product to buy without naming yours.

02 / Work the numbers

The false win you can reproduce

Imagine a small, fixed panel of twelve questions on one named product surface, with one planned check per question in each period. This is deliberately small enough to inspect, not a recommended sample size for estimating a market.

In Period A, ten checks return usable answers. Four mention the brand. In Period B, only six return usable answers, including the same four mentioning it. Four questions that previously returned non-mentions are now missing. Two checks are missing in both periods.

Among completed answers40% → 67%4/10 → 4/6
Collection completion83% → 50%10/12 → 6/12
Observed mentions4 → 4No additional mention recorded

The percentage increase is real arithmetic. The claim that the brand improved does not follow from it. In this example, excluding the newly missing checks explains the increase. We do not know what those checks would have returned.

Now compare only the six questions completed in both periods. The result is four mentions out of six in each: about 67% versus 67%. That is a helpful diagnostic, but it still leaves half of the planned panel out.

You can also state how much the missing answers leave unresolved. In A, the twelve planned checks could contain between four and six mentions, depending on the two missing outcomes: 33% to 50%. In B, the range is four to ten: 33% to 83%. These are missing-outcome bounds for this invented panel, not statistical confidence intervals and not estimates of all AI answers.

The ranges overlap. More importantly, the later measurement is less complete. The responsible conclusion is not that performance fell or rose. It is that this collection cannot establish improvement.

Inspect all 12 rows and formulas

03 / Check comparability

Freeze the measurement, not your business

Before the next report, write a short measurement contract. It can fit in a spreadsheet header:

Decision: whether our new comparison page is appearing in a defined set of buyer answers.

Panel: stable question IDs and exact wording; buyer intent, language and market; branded and unbranded groups kept separate.

Collection: named engine and product surface, API or consumer interface, visible search settings, dates, time zone, repeat rule and retry policy.

Scoring: mention, suitable recommendation and owned-domain link recorded separately; unknown and failed checks retained.

Changes: version any question, market, collection or classification change before joining periods into a trend.

The surface matters. An API answer and a consumer chat answer are not automatically the same observation. OpenAI says location and relevant saved memories can affect search-query rewriting. Google says AI Mode and AI Overviews may use different models and techniques, so their answers and links can differ. Record the conditions you can observe and label the ones you cannot. [2] [1]

Keep a stable core panel for trend checks and a separate discovery panel for new buyer questions. Your market can change without making every week a new, incomparable test. Report the two panels separately until you have a comparable baseline.

Repeat collection according to a rule set before seeing results. Do not rerun only the answers you dislike. Extra runs reveal variation, but closely related questions and repeated model outputs are not automatically independent observations. A handful of repeats cannot justify a precise market-wide confidence claim.

04 / Inspect the evidence

Ask for an export before another subscription

You do not need a second dashboard to discover whether the first can explain its own number. Ask for answer-level evidence for the report you already have.

Recalculate one headline metric

Request stable question IDs, exact prompts, period, engine and surface, market, timestamp, completion status, full answer, returned links and scoring labels. Preserve attempts and retries separately, but use the predeclared rule to decide which attempt enters the score. A retry is not a new buyer.

Read the failures, not only the winners

Separate an answer that omits your brand from a timeout, a truncated response, a refusal and missing citation metadata. Missing metadata does not prove that the original answer had no citations. A zero is a finding; an unknown is an evidence gap.

Recode before looking at the label

Read a small, varied set of raw answers without the dashboard verdict. Include one apparent win, one omission and one ambiguous case. Check entity aliases, negative mentions and whether a recommendation actually fits the buyer. This is a spot-check for errors, not validation of the whole dataset.

Make the tool explain a disagreement

A different score may come from different questions, collection, entity matching, weighting or missingness. Find the layer where the disagreement starts. Do not pick the higher score because it is more encouraging.

These principles are not new inventions. AIBL's proof-of-method guide and Llamat's checklist already make detailed cases for inspecting evidence, failures and collection methods. The worksheet here makes one failure mode visible row by row, so you can reproduce it rather than take another checklist on trust. Both publishers sell tools; apply the same questions to them. [5] [6]

KEEP THE EVIDENCE CONNECTED

01QuestionSame wording
Same buyer + market
02AnswerFull response
Or explicit missing
03OutcomeMention ≠ recommendation
Link ≠ visit
Reconstruct the score from observations before using it to choose the next action.

05 / Decide what to do

Fix the measurement, then choose the intervention

What the evidence supports next

Completion changed

First check
Failures by topic, engine and market; retry policy; collection changes.
Decision boundary
Repair collection before crediting content or changing strategy.

Comparable panel improved

First check
Raw answers, recommendation quality, variation and any model or classification change.
Decision boundary
Report a bounded improvement in the tested panel, not market dominance or causality.

Links rose, visits did not

First check
The cited URLs, supported claims, referral identification and landing pages.
Decision boundary
Do not call citations traffic, or assume the panel represents all users.

Visits rose, conversions did not

First check
Tracking, landing-page fit, signup friction and the definition of a qualified lead.
Decision boundary
Do not buy more visibility to solve an unexamined conversion problem.

Different first-party reports also answer different questions. Bing's AI Performance documentation describes citations across supported Microsoft and partner surfaces; it says grounding queries are sampled and citations do not indicate placement. Google now documents a dedicated Generative AI performance report for AI Overviews and AI Mode impressions, also included in overall Web performance; the report may not appear when a site has insufficient impressions. Neither report is a universal census of every assistant conversation. [3] [11]

Referral analytics is another layer. Plausible, for example, describes its AI channel as visits and on-site outcomes, rather than mentions in a sampled prompt panel. That makes it useful for a different decision, not inherently a better measure of the same thing. Attribution is limited to what the implementation can identify and record. [4]

Do not divide analytics visits by mentions in your test panel to invent an AI click-through rate. The visits and sampled answers are not the same population. Likewise, a better matched-panel score after a content edit is an association, not proof that your edit caused the change.

06 / Report it honestly

A useful update can be four sentences

We observed four brand mentions in each period. Collection completion fell from ten of twelve checks to six of twelve, so mention coverage among completed answers rose from 40% to 67%. On the six checks completed in both periods, coverage was unchanged at four of six. We cannot claim improvement from this report; we will investigate the missing checks and keep referral visits and conversions in a separate report.

That is less exciting than a large green arrow. It is more useful to the person deciding where the next month of effort goes.

When choosing a tool, pay for coverage and workflow you can explain: the buyer questions you care about, the surfaces and markets you need, usable raw evidence, visible failure handling and an export you can recalculate. Engine count and a polished scorecard are not substitutes.

The buying rule is simple: if a vendor cannot show what changed underneath its score, do not use that score to justify more spend. Keep the tool for narrower tasks if it genuinely helps, but narrow the claim with it.

Method and sources

What this article did, and did not test

This is a proposed audit method with a fully invented teaching dataset. No AI visibility platform was performance-tested, no customer results are claimed, and no live prompts were run for the example. Twelve rows make the arithmetic inspectable; they are not a recommended research sample size.

The linked r/SEO discussion already gives an invented denominator-change example and proposes separate mention/completion/failure counts, comparison of common answered questions, failures by topic, distinct mention/citation/recommendation fields and dated raw answers. Those ideas are prior work, not a new method introduced here. This article develops a different twelve-row teaching example, explicit missing-outcome bounds, a CSV and a reporting template to make the problem reproducible. [7]

This article recommends no paid product and makes no ranking claim. Product documentation and reporting details linked in this article were read on 18 November 2026; they can change.

  1. Google Search Central: AI features and your website

    Requirements, search-surface differences and Search Console reporting.

  2. OpenAI: Searching the web with ChatGPT

    Search, citation links, location and memory context.

  3. Bing: Introducing AI Performance

    Publisher citation metrics, supported surfaces and sampled grounding queries.

  4. Plausible: AI traffic analytics

    Referral visits and conversions versus sampled answer visibility. Analytics vendor, not an independent accuracy benchmark.

  5. AIBL: How to evaluate AI visibility tools

    Prior work on frozen trials, failures, evidence exports and comparable denominators. Tool vendor.

  6. Llamat: AI visibility tool evaluation checklist

    Prior work on collection methods, repetition and evidence. Tool vendor.

  7. r/SEO: Separating improvement from sample change

    Prior practitioner example of denominator drift, separate completion and mention counts, common answered questions, failures by topic and distinct outcome fields. Not a prevalence study.

  8. Obi: product overview and FAQ

    Vendor description of software-startup buyer questions and approval-gated website drafts.

  9. AirOps: content engineering teams

    Vendor description of content operations, governance and workflow roles.

  10. Profound: enterprise capabilities

    Vendor description of enterprise controls including SSO and role-based access.

  11. Google Search Console: Generative AI performance report

    Current dedicated AI Overviews and AI Mode impression reporting, availability threshold and relationship to Web reporting.

The GEO Directory. All percentages in the worked example are calculated from invented records, not market observations.

Invented data · 12 planned checks per period

The complete audit worksheet

Every number in the article can be rebuilt from these rows. Missing means the outcome is unknown, not that the brand was absent.

Back to the article

Full ledger

One row per planned check

Q01A: MentionB: Mention
Q02A: MentionB: Mention
Q03A: MentionB: Mention
Q04A: MentionB: Mention
Q05A: No mentionB: No mention
Q06A: No mentionB: No mention
Q07A: No mentionB: Missing
Q08A: No mentionB: Missing
Q09A: No mentionB: Missing
Q10A: No mentionB: Missing
Q11A: MissingB: Missing
Q12A: MissingB: Missing
View and copy the invented CSV
question_id,period_a_status,period_a_mention,period_b_status,period_b_mention
Q01,complete,yes,complete,yes
Q02,complete,yes,complete,yes
Q03,complete,yes,complete,yes
Q04,complete,yes,complete,yes
Q05,complete,no,complete,no
Q06,complete,no,complete,no
Q07,complete,no,missing,unknown
Q08,complete,no,missing,unknown
Q09,complete,no,missing,unknown
Q10,complete,no,missing,unknown
Q11,missing,unknown,missing,unknown
Q12,missing,unknown,missing,unknown

No live measurements, brand names or vendor results are in this file. One check per question per period, on one hypothetical surface.

Recalculate

Counts before percentages

Same twelve planned question IDs in both periods

Period A

Planned
12
Completed
10
Missing
2
Mentions observed
4
Confirmed non-mentions
6
Completion
10/12 = 83.3%
Mention rate among completed
4/10 = 40.0%
Common completed subset
Q01-Q06: 4/6 = 66.7%
Possible mentions in planned panel
4 to 6 of 12 = 33.3%-50.0%

Period B

Planned
12
Completed
6
Missing
6
Mentions observed
4
Confirmed non-mentions
2
Completion
6/12 = 50.0%
Mention rate among completed
4/6 = 66.7%
Common completed subset
Q01-Q06: 4/6 = 66.7%
Possible mentions in planned panel
4 to 10 of 12 = 33.3%-83.3%

Completion = completed checks / planned checks.

Mention rate among completed = answers mentioning the brand / completed usable answers. This definition applies only to this example; check the vendor's actual metric.

Missing-outcome bounds = observed mentions / planned checks, up to (observed mentions + missing outcomes) / planned checks. This asks what the fixed panel could contain if missing outcomes were all non-mentions or all mentions. It does not infer the missing answers or account for model variation.

Use with your own data

Keep an evidence trail

Extend the CSV with prompt text and version, intent, brand aliases, engine and surface, market, language, timestamp and time zone, attempt ID, status, retry relationship, raw-answer reference, recommendation label, owned citation URL, reviewer and scoring-rule version. Use an explicit unknown value when evidence is missing. Do not fill blanks with zeros.

Keep a record for every planned check. Decide how retries enter the score before collecting. If you change the panel or collection method, mark the break and report comparable subsets alongside completion, not instead of it.

Use this to audit arithmetic and missingness. It does not validate a vendor, establish causal impact or replace a measurement plan suited to your decision.