Googlebot and AI crawler analytics: see which bots read your site

By Andrei Sheshenin · October 19, 2026

ClickBeagle shows which search and AI crawlers read your site, including Googlebot, GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot and Bingbot, which pages they request and which requests fail. It reads your server or Cloudflare logs, not the JavaScript counter, and checks each bot against the operator's published IP ranges or DNS so fake “Googlebots” are not counted as real ones.

Why regular analytics can't see crawlers

Tools like GA4 run in the visitor's browser. Crawlers usually fetch the HTML without running your analytics script, and GA4 deliberately excludes known bot traffic from reports. So the question “is ChatGPT reading my service pages?” cannot be answered from browser analytics. It needs request logs from the server or the CDN in front of it.

Where the data comes from

• Cloudflare: if your domain is proxied by Cloudflare, ClickBeagle pulls crawler requests for your hostnames every hour. Cloudflare's data is sampled, so each row carries its weight and the report says the coverage is sampled. The first pull looks back 3 days, which is what Cloudflare's free plan keeps.
• Access log upload: upload a standard access log (Combined Log Format) from your host in the dashboard.
• HTTP logs API: a signed endpoint for hosts and proxies that can forward request records.

Only requests from declared bots and generic automation are stored. Human requests in the logs are counted per source but not kept.

Which bots ClickBeagle recognizes

Other user agents that look like automation (generic “bot”, crawler, curl, python-requests, headless browsers) are grouped as unidentified automation. Google-Extended is a robots.txt token, not a crawler, so it never appears as a bot of its own.

Verification: why the user agent is not enough

Anyone can send a request that says “Googlebot”. ClickBeagle first checks the request IP against the lists the operators publish (Google, OpenAI, Perplexity, Bing), then falls back to reverse DNS with a forward confirmation (for example, the IP must resolve to a googlebot.com host that resolves back to the same IP). It does not verify bots by network owner alone, and it never trusts a forwarded-for header from a direct client. A request that claims to be a bot but comes from outside its published ranges is marked as a mismatch. Bots whose operator publishes no IP list or DNS method are shown as declared by user agent, not verified.

What to look for

• AI search crawlers on money pages. If OAI-SearchBot, Claude-SearchBot or PerplexityBot never request your service and area pages, AI assistants have less to cite when someone asks for a local provider.
• User-triggered fetchers. ChatGPT-User, Claude-User and Perplexity-User requests usually mean a person asked an assistant about a page. They are a demand signal, not a crawl.
• Training crawlers versus search crawlers. GPTBot, ClaudeBot, CCBot and Bytespider collect training data; blocking them in robots.txt is a separate decision from blocking AI search crawlers.
• Errors. 404s and 5xx responses served to Googlebot or AI crawlers, and whether they fetch robots.txt and your sitemap.

Deciding what to allow in robots.txt

Once you see who crawls you, robots.txt is where you decide. The operators document separate tokens, so you can allow AI search while declining model training, for example:

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

Blocking a training crawler does not remove you from that company's search features, and blocking an AI search crawler makes it less likely an assistant can cite your pages. For Google, the Google-Extended token controls use of your content for Gemini models without affecting Google Search. After a change, the bot report shows whether each crawler actually follows it: requests to blocked paths should stop within days, and a crawler that keeps coming is worth a closer look at its verification status.

Bots read, people arrive: AI referrals

Crawling is half the picture. ClickBeagle's regular visit reports label visitors who arrive from chatgpt.com, perplexity.ai, claude.ai, Gemini or Copilot as AI-assistant traffic, so you can see whether being read turns into visits and leads. You can ask both questions through the MCP server with the bots_activity and list_visits tools: “Does ChatGPT crawl my site, and did anyone come from ChatGPT this month?”

For the full picture of what ClickBeagle combines, see the features or our analytics tools comparison.

Related articles