How to Detect AI Agents Using or Citing My Content
Crawler fetches, AI citations and AI referral clicks need different handling. What to automate, like error-spike alerts and a weekly crawl-to-citation review, and what to check by hand.
Detecting AI use of your content is three jobs, not one. Bots fetch your pages, AI answers cite them, and people click through from those answers. Each leaves evidence somewhere different. The useful question for a small team is which of those jobs can run without anyone watching, and which ones still need a person to look.
Our rule of thumb: automate anything that is collection or a threshold, and keep a human on anything that is a judgment about identity, policy or meaning.
The three signals
Fetches show up in server or CDN logs, identified by user agent. OpenAI's bot docs list OAI-SearchBot (ChatGPT search), GPTBot (training) and ChatGPT-User (visits a user triggered in ChatGPT or a Custom GPT). Anthropic's crawler page lists ClaudeBot (training), Claude-SearchBot (search indexing) and Claude-User (fetches when a user asks). Perplexity's bot guide lists PerplexityBot (search, not training) and Perplexity-User (user-requested fetches). Google lists GoogleOther as a generic crawler for its product teams, and notes that Google-Extended has no user agent at all: it's a robots.txt token for Gemini training and grounding (Google's crawler docs).
Citations happen inside the AI answer. You only see them by running prompts and recording which URLs each answer cites.
Clicks show up in analytics. OpenAI's publisher FAQ says ChatGPT appends utm_source=chatgpt.com to referral URLs.
| Signal | Where to look | What it proves | Blind spot | Automate or inspect |
|---|---|---|---|---|
| Fetch | CDN or server logs, by user agent and IP | A bot reached the page | Whether any answer used it | Automate collection and alerting; inspect identity oddities |
| Citation | Recorded answers to tracked prompts | The engine used your URL as a source | Prompts you don't track; clicks | Automate the prompt runs; read answers by hand when something moves |
| Click | Referrers, utm_source=chatgpt.com | A person arrived from an AI answer | Citations nobody clicked; the prompt behind the visit | Automate; inspect only when it disagrees with citations |
A crawl isn't a citation. A citation with no click leaves nothing in analytics. Any automation you build has to keep those apart, or it will report the wrong thing very efficiently.
What to automate
Log ingestion. Set this up once and never touch it again. Promptwatch Agent Analytics pulls real-time AI crawler logs from Cloudflare, AWS CloudFront, Fastly, Vercel, Netlify, Akamai, Google Cloud CDN or a custom HTTP endpoint. Pulling from the edge matters, because origin logs miss whatever the CDN rejected before it reached your server.
Classification and verification. Matching a request against a vendor's published IP list is a rule, so a machine should apply it. OpenAI publishes openai.com/searchbot.json, openai.com/gptbot.json and openai.com/chatgpt-user.json. Perplexity publishes perplexity.com/perplexitybot.json and perplexity.com/perplexity-user.json, and recommends combining user agent and IP matching in WAF rules. Google recommends published IP ranges or reverse DNS. Anthropic notes that IP blocking may not work reliably for its bots. Whatever tool you use should sort requests into verified, unverified and unknown without asking you.
Alerts on error spikes. A jump in 403s or 5xx responses to OAI-SearchBot or PerplexityBot is the one fetch event worth interrupting someone for, because it can quietly take pages out of AI search. Where the alert fires depends on your stack. Known Agents lists spike alerts and error and redirect analysis among its features. Cloudflare AI Crawl Control shows status code distribution per crawler on every Cloudflare plan, though its listing doesn't mention alerting. In Promptwatch, error tracking in Agent Analytics is where you diagnose the spike by crawler and page. We haven't seen a documented push alert for it, so if you want a ping, wire it from the edge tool or a script you own.
Prompt runs and citation capture. Nobody should be pasting prompts into ChatGPT by hand. Paid Promptwatch plans refresh tracked prompts daily and record citations at page and domain level, with Reddit and YouTube citations separated out.
A weekly crawl-to-citation review, drafted by an agent. This is where an agent saves the most time. With a read-only key on the Promptwatch MCP server, an agent can pull getCrawlerTrend and getTopCrawlerPages for the fetch side, getCitations and getCitationTopPages for citations, and getVisitorReferrerStats and getVisitorTopPages for clicks. From those it writes a short note: pages crawled but never cited, pages cited but not recently crawled, cited pages that send no visits. Run it as a saved prompt in Claude every Monday, or ask the Promptwatch Slack agent the same questions in your team channel. Keeping the weekly cadence is on you. The pulling and drafting are not.
What to inspect by hand
Unverified spikes. A surge of requests that claim to be GPTBot but fail the IP check could be a scraper or a misconfigured monitor. Deciding whether to block it is a security call.
Robots.txt policy. Whether to allow training crawlers is a business decision, separate from whether you want to appear in AI search. Know the rules before you write them. OpenAI says robots.txt may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores it. Anthropic says ClaudeBot, Claude-User and Claude-SearchBot all respect it, and each must be blocked separately per subdomain. An agent can draft the change. A person should ship it.
The answers themselves. When a page gains or loses a citation, read the answer. Is your page cited for the claim you want, or for a stray paragraph? Is a competitor cited next to you for the same point? No metric captures that.
Referrals that don't match citations. If utm_source=chatgpt.com visits land on a page that your tracked prompts never cite, people are asking prompts you don't track. That's a gap in your prompt list, and only a person can decide which new prompts to add.
Judging tools by what they take off your plate
Promptwatch: monitors all three signals, recommends, and executes. It joins fetches, citations and conversions per URL through the crawl-to-citation path and visitor analytics, which install by script or GTM template. Unified Actions turns the findings into a to-do list, and content gap analysis shows which prompts have no matching page. When the weekly review finds a page that is crawled but never cited, Content Agents can draft the fix and publish it to Webflow, Framer or WordPress through a review inbox. Professional at $245/mo includes 25M crawler logs, 150 prompts, 1M visitor events and 5 AEO articles; the automated Content Agent is on Business ($579/mo).
Known Agents: monitors fetches and clicks, and acts on access. Automatic Robots.txt adds rules for chosen bot categories as new agents appear. Its citation counts are estimates inferred from agent fetches, per its own llms.txt, so it can't tell you what an answer actually cited.
Cloudflare AI Crawl Control: monitors fetches and enforces blocks. Allowing or blocking a crawler writes a WAF rule. It's free on every Cloudflare plan, but detection is user agent only on the Free plan, and it has no prompt or citation data.
For a worked example of the access side, see how an agent can read crawler logs and propose robots.txt fixes. Compare more tools on our agentic SEO rankings, or start with promptwatch.com.