From Prompt to Crawler to Response: Tracing the Full Lifecycle of an AI Query

Executive Takeaway

What This Study Does and Does Not Prove

This study measures observed fetch behavior (which pages AI platform crawlers requested, when, and from where) and citation outcomes (which URLs appeared in AI-generated responses) for a single domain across a single 63-minute study window using 27 prompts.

All findings are scoped to this dataset. Where we interpret patterns or suggest mechanisms, we flag those as hypotheses.

Findings

What follows is a set of observations from this study — interesting findings in AI agent/crawler behavior and how crawled data showed up in answers. We focus on the journey from prompt to crawler to pages hit, latency, size of content retrieved (where visible), and how the response and citations reflected (or didn't reflect) what was fetched.

Structured Hub Pages as Citation Multipliers

The single highest-citation crawl event in the study:

ChatGPT fetched the /intelligence index page — a single HTML document listing published research articles with internal links. From that one page crawl, it generated 12 citation instances across 11 distinct URLs on our domain (shown in the flow above).

We did not observe ChatGPT fetching any of those articles individually. The behavior is consistent with the model parsing the index page's internal links and citing discovered URLs without following them — though we cannot confirm this is the mechanism, only that no additional fetches were logged.

Across all platforms in this dataset, the /intelligence index page was cited 20 times — more than any individual article. Compare with the url-llms probe, where ChatGPT fetched a single flat-text file and produced zero citations. Here, structured HTML with internal link architecture correlated with far more citations than plain text.

Distinct Fetch-to-Cite Architectures Across Platforms

From our telemetry we can describe response patterns — e.g. "we observed live crawl/fetch here," "we observed no site-side fetch here," "the response looked consistent with indexed or prior knowledge" — but we generally cannot prove a provider's full internal architecture, that "no observed crawl" means no retrieval happened, or that one vendor is always live-fetch and another always cache-only. Below we frame three patterns that fit what we saw, with links to vendor documentation where it clarifies what is possible.

Citation behavior for platforms that in this run showed no observable site-side fetch:

Observation: In responses where we saw no live fetch, content that appears to have been in training or indexing pipelines was still cited; content that may exist only on the live site (e.g. case studies, FAQ) was rarely cited. We do not infer that these providers lack web retrieval — only that we did not observe it in this run.

Quantitative Research Titles Correlated with Higher Citation Rates

Not all intelligence articles performed equally. In this dataset, articles with specific, quantitative framing in the title received substantially more citations.

The top performers share two observable characteristics: (1) Specific quantitative language in the URL — "quantify," "index," "revenue" signal measurable data. (2) Multi-platform crawl interest — the corporate-spend article was independently crawled by ChatGPT, Perplexity, AND Google (via Nexus 5X render pass at 23:10 UTC). The lowest performer (geo-isnt-seo-with-a-new-acronym) uses conceptual/opinion framing and was not crawled by any platform during the observation window.

AI-Readiness Content Attracted Strong Citation Behavior

The prompt "Does Second Wind have a dedicated AI-readable version of their site?" achieved 5/5 citation rate — every platform cited our domain. This was the only non-URL, non-comparison prompt to achieve a perfect score in our dataset. Every platform: acknowledged the AI Surface concept; used the phrase "AI-readable" in its response; Grok specifically cited ai.secondwind.cloud; Claude cited us on this prompt despite producing no citations on 18 of 27 others. This was also the prompt that generated the most owned citations from Claude (2) and Grok (7) — in this run we observed no site-side fetch for those platforms, though both document web search or retrieval capabilities. Interpretation: This finding is consistent with the idea that AI systems have a natural affinity for content describing AI-readable infrastructure — it directly relates to their own data ingestion paradigm. However, we note that this prompt is also highly specific and brand-relevant, which may independently explain the high citation rate. We cannot isolate which factor drives the result.

Illustrative Traces

Three representative prompt-to-crawler-to-response examples. Below we map the full path from prompt → crawler → pages hit → latency → what we infer was retrieved → response and citations. These three journeys use only events we directly observed; they show how crawled data did or didn't show up in the answer.

Detailed Data

Probe Configuration

27 prompts across 8 families: brand (3), intelligence (5), case-study (2), surface/competitive (5), url-explicit (4), evaluation (3), use-case (3), trust (2). Each fired sequentially across all 5 platforms.

Mention & Citation Rates

Overall: 100% mention rate (135/135) · 67% citation rate (91/135) · 23.8% owned citation share (228/958)

Pages Cited by Platform

Explicit URL Probes vs. Organic Discovery

Avg owned cites per probe: Explicit URL 2.4, Organic 1.6. ChatGPT is the only platform where organic probes produced a higher citation rate than explicit URL probes. Claude saw the largest boost from explicit URLs: citation rate nearly tripled from 26% to 75%.

Citation Grid — Per Prompt × Platform

CITE = cited our URL · MENT = mentioned but no citation. Perfect scores (5/5): surface-organic, url-homepage, url-surface, eval-compare, usecase-finserv, trust-privacy. Zero citations (0/5): eval-faq, trust-legitimacy.

Data Volume

Methodology

• Probe design: 27 prompts across 8 families fired sequentially across 5 platforms (135 total probes).
• Crawl observation: 15-second observation window after each probe, supplemented by post-hoc telemetry correlation for the full study duration and 3-hour post-study window.
• Crawler detection: Real-time middleware using UA string matching, behavioral fingerprinting, and IP verification. Each event assigned a confidence score and match method.
• Study duration: 63 minutes active (20:27–21:31 UTC), post-study observation through 00:28 UTC.
• Success rate: All 135 probes completed successfully.
• Entity confusion: Manual review of all 135 response texts, flagging responses that described entities other than the target company.

Limitations

• Single domain; results may not generalize.
• Single 63-minute session — a snapshot, not longitudinal behavior.
• 16 observed crawl events; statistical significance would require larger samples.
• Correlation, not causation — we cannot confirm that probes caused crawls or that crawls caused citation patterns.
• Telemetry captures requests to our domain only; platforms may fetch other sources we cannot observe.
• Results are specific to the model versions tested and may not reflect newer or different versions.

Practical Takeaways for Site Operators

1. Build structured hub pages that expose citable internal links. In this study, a single index page with clear link architecture and structured data (e.g. CollectionPage, ItemList) generated more citations than any individual article. Structure matters as much as content.
2. Use llms.txt for representation, not as a citation strategy. The data suggests llms.txt shapes how AI systems describe you — which has value — but does not generate citation backlinks. Treat it as a context layer, not a link-building tool.
3. Publish domain-specific content that disambiguates your entity. Responses that drew on unique, specific content (research articles, quantitative analysis) showed less entity confusion than those relying on generic homepage copy.
4. Don't assume uniform behavior across platforms. In this study we observed: ChatGPT fetching pages in real time and appearing to use internal links from a single fetch; Perplexity drawing on a search index and sometimes fetching at query time; Grok, Claude, and Gemini citing without any observed site-side crawl in our window — though all document web search or retrieval. A strategy optimized for one pattern may underperform on another.
5. Quantitative, specific framing correlates with higher citation rates. Articles with measurable claims in the title outperformed conceptual or opinion-framed pieces in this dataset.