Citation Ledger methodology

How the citation data is collected, classified, compared, and audited. This page documents the decisions behind the numbers and the places where the data remains incomplete.

Current wave: 1,500 queries · 50 of 50 metros · September 2026

What the study measures

When Gemini answers a question such as “Who is a good plumber in Glendale, Arizona?” it can ground the response in web sources. The Citation Ledger records those sources and the businesses the answer recommends.

There is no published ranking formula for local-business citations in AI answers. This project measures the returned answers directly instead of trying to infer a formula from isolated examples.

Primary data source

The main study uses live calls to the Gemini API with Google Search grounding. It does not scrape a search-results page or rely on manually selected examples.

The URI inside grounding_chunks is a Google redirect rather than the source URL. The accompanying title field contains the cited domain, such as bbb.org or reddit.com. Domain-level analysis uses that title field. We checked it against live responses before collection.

Query design

Metros

The study uses the 50 largest US Metropolitan Statistical Areas by Census-estimated population. Each MSA is represented by its principal city and state so the queries read naturally. Phoenix–Mesa–Chandler, for example, becomes Phoenix, Arizona.

Verticals

The ten categories span emergency trades, professional services, and everyday local services:

Plumber · Roofer · HVAC contractor · Electrician · Locksmith · Pest control · House cleaning · Personal injury lawyer · Dentist · Auto repair shop

Query templates

Each metro-and-vertical combination uses three phrasings:

• best {vertical} in {city}, {state}
• top rated {vertical} near {city}, {state}
• who is a good {vertical} in {city}

The complete matrix is 50 metros × 10 verticals × 3 phrasings: 1,500 queries per full pass.

The July baseline returned 1,487 usable Gemini queries. Thirteen calls failed every retry and were skipped as technical failures, not sampling choices. The published page’s separate claim that 498 of 500 groups had all three phrasings is mathematically incompatible with 1,487 successful queries, so that group count is not repeated here pending verification from the raw rows. The September wave completed all 1,500 queries.

Citation taxonomy

Each cited domain is assigned to one of nine current categories:

The original July taxonomy had eight categories; Google Maps was added for cross-engine work and historical waves were restated under the audited current taxonomy.

Domains are checked rather than classified from their names alone. For example, serviceagent.ai is vendor marketing content, not a directory. gaf.com is a roofing-material manufacturer and certified-installer finder, not an individual roofer’s website.

How the fallback category is audited

Any domain without a specific taxonomy rule defaults to “business’s own site.” That works for much of the local-service long tail, but it can also hide aggregators and publishers.

An early audit checked the 15 highest-frequency unrecognized domains and four randomly selected low-frequency domains.

• Nine of the 15 high-frequency defaults—60%—were misclassified.
• All four low-frequency checks were genuine individual businesses.

The nine corrections were phillymag.com, washingtonian.com, repairpal.com, opencare.com, roofpitch.net, pestcontrolcalifornia.us, birdeye.com, aaa.com, and eloa.org. They included city magazines, lead-generation marketplaces, vendor content, and membership directories. At that point, the headline business-site share moved from 61.1% to 58.7%.

The current process prints every unrecognized domain responsible for more than 0.5% of a wave’s citations. Each is visited and classified manually. The September audit reclassified roughly three dozen domains, including reviews.birdeye.com, bestprosintown.com, avvo.com, and attorneys.superlawyers.com.

After that refresh, the audit list was empty for every wave. This does not prove that every low-frequency domain is correct; it limits the largest known source of aggregate error.

Extracting recommended business names

Business names are parsed from bold text at the beginning of numbered or bulleted answer items. Pattern filters and a short stoplist remove subheadings such as “Highlights” and generic labels such as “Top Local Companies.” Both were checked against sample answers before full collection.

The extractor recognizes business names in roughly 70–75% of queries. The remaining answers use prose structures it does not recognize.

September added a filter for imperative advice bullets such as “Get multiple quotes,” which the earlier extractor could mistake for company names. Disabling that filter still leaves September’s no-recognizable-business rate at 45.3%, compared with 25.8% in July. The filter explains about 31 of 328 additional unmatched queries; it does not explain the overall change.

Google Places cross-reference

Extracted business names are matched to Google Places Text Search using the business name and metro. The lookup supplies star rating, review count, and address. Results are cached by business name and metro.

Control group for ratings

For the same 50 metros and 10 verticals, the control group uses a plain Google Places category query such as “plumbers in Phoenix, Arizona.” It contains no “best” or ranking language. Every returned result—up to 20 per cell—is retained whether or not Gemini cited it.

This is not a random sample of every local business. Google’s category search already favors businesses its system surfaces. The appropriate description is “businesses returned by a plain Google Places search,” not “all businesses.”

Measuring phrasing volatility

For every metro-and-vertical group, the cited-domain sets from the three phrasings are compared pairwise with Jaccard similarity:

intersection ÷ union

The three pair scores are then averaged. A score of 1 means the phrasings cited identical domain sets. A score near zero means little overlap.

Confidence intervals and significance tests

Headline citation shares are bootstrapped at the query level with 2,000 resamples. Each resample draws queries with replacement and recomputes the share.

Queries—not individual citations—are the sampling unit because the eight to ten citations returned inside one answer are not independent observations.

The 0.08–0.10-star rating-gap interval uses a separate business-level bootstrap. Recommended and control businesses are resampled independently because the comparison is between two business sets.

Vertical differences in phrasing volatility use a two-sided permutation test with 2,000 shuffles. The test compares group-level Jaccard scores and counts shuffled differences at least as extreme as the observed one.

The published extreme-vertical comparison was selected after seeing which of the ten verticals were highest and lowest. That introduces selection bias. A strict correction across every possible pair would require a substantially smaller p-value. Treat the ordering as a pattern, not a precisely established ranking of all ten verticals.

Local-pack control test

The control test asks whether Grounding Drift is specific to AI answers or merely reflects ordinary local-search movement.

It uses the same 500 metro-and-vertical combinations as the main study, one fixed phrasing per combination, and five completed rounds spaced five minutes apart.

One arm uses Gemini. The other uses Google’s classic local pack for the identical query through DataForSEO’s Live Google Organic Advanced endpoint. A commercial provider was used because automated direct scraping of Google results would violate Google’s terms.

Business names come from DataForSEO’s standalone local_pack result items. Consistency is measured in two ways:

• Strict: whether the top-listed business matches between rounds
• Loose: Jaccard overlap across the complete business lists

The complete five-round control retains the pattern seen after the first two rounds: Gemini’s returned businesses move far more than the local pack’s.

Gemini versus ChatGPT

The same 1,500-query design was also collected from ChatGPT Search. The metros, verticals, and literal query text matched the Gemini dataset.

ChatGPT results came through DataForSEO’s AI Optimization/ChatGPT LLM Scraper endpoint: ai_optimization/chat_gpt/llm_scraper/live/advanced. It reads the consumer-facing Search product rather than the raw OpenAI API. A commercial provider was used instead of directly scraping the product.

The endpoint supports country-level location only. Every request used “United States,” with city and state included in the query text. Collection ran with ten concurrent requests; the endpoint averaged about 19 seconds per response. DataForSEO permits up to 30 simultaneous live requests per platform.

Before analysis, leading www. prefixes were removed from ChatGPT domains. Without that normalization, www.reddit.com and reddit.com would be treated as different sources, known domains would fall into the business-site fallback, and cross-engine matches would be missed.

All 1,500 ChatGPT queries completed, all 1,500 matched Gemini keys, and the logged cost was $6.00.

Citation categories

The mean Jaccard overlap between the engines’ cited-domain sets was 8.3% across 1,500 matched queries.

Gemini returns roughly eight to ten citations per query. ChatGPT averages about 1.9. The Jaccard comparison does not adjust for that difference in set size, so it is a description of the returned sets—not a size-normalized similarity score.

Recommended businesses

Both engines named at least one recognizable business on 783 queries. On those queries:

• The top business matched 4.9% of the time.
• Mean Jaccard overlap across the complete recommendation lists was 0.052.

Gemini names were extracted with the study’s regex process. ChatGPT names came from DataForSEO’s structured brand_entities field.

The number of queries where both engines named a recognizable business fell from 1,099 in July to 783 in September—about 29%.

Gemini named no recognizable business on 47.4% of September queries, up from 25.8% in July. ChatGPT changed from 0.4% to 0.9%. Even with the new Gemini advice-bullet filter disabled, September remained at 45.3%.

That is a measured change in returned answer structure. It coincided with fewer citations per query, a narrower source pool, more directories, and fewer named companies; the study does not establish a single cause.

Gemini fan-out queries versus People Also Ask

Gemini grounding records the internal search strings it generates in web_search_queries. We compared them with Google’s People Also Ask questions for the identical 1,500 seed queries.

Lexical overlap was scored from zero to one:

• Average overlap: 15.2%
• Gemini fan-out queries scoring below 0.2: 73%
• Highest vertical: Pest control, 26.6%
• Lowest vertical: Auto repair, 6.4%

This measures query wording inside the grounding process. It is not a citation-visibility result and does not support an optimization recommendation by itself.

Known limitations

Each wave is a snapshot

July is the frozen baseline. The identical design is rerun as separately dated waves; previous numbers are not overwritten.

Identical queries disagree on citations roughly 60% of the time. Small wave-to-wave changes should therefore be treated cautiously. The What Changed page reports movements large enough to stand above the observed repeat-run variability.

The main study is Gemini-centric

The recurring 1,500-query taxonomy and volatility measures use Gemini API grounding. ChatGPT appears in a separate single-pass comparison. Perplexity, Microsoft Copilot, and Google AI Overviews are outside this build.

API results may differ from consumer products

gemini-flash-latest with API grounding may retrieve and rank sources differently from the Gemini consumer app or Google’s AI Overviews surface.

The ChatGPT collection uses a third-party scraper of the consumer-facing Search mode, with country-level—not cit

See the stats/study in context: Steady Demand Research Index