- AI crawlers are largely invisible to Google Analytics/GA4 because most don't execute JavaScript. The tracking script never fires. Raw server access logs are the only reliable source for this traffic.
- A hit count alone isn't enough: the response code matters. A 200 means the bot actually got your content; a 403 or 429 means it was blocked or throttled, and a raw count doesn't distinguish the two.
- Which specific paths get crawled (and which get ignored) tells you more than a site-wide total: a bot hitting your homepage 40 times and never touching your actual content pages is a different problem than low traffic overall.
- A single day's log is a noisy sample. Build a recurring check (weekly or monthly) rather than treating one audit as the answer.
- The entire practical workflow here is grep, awk, and a cron job. No paid tool required, though Screaming Frog's Log File Analyser (free to 1,000 events) and Cloudflare's native AI Crawl Control dashboard are the two standard named references once you want more than that.
Standard analytics tools have a blind spot that matters a lot for this specific question: most AI crawlers request raw HTML and parse it directly, without executing the JavaScript that fires your analytics tracking script. GA4 never sees them. Server access logs, which record every request at the infrastructure level regardless of whether JavaScript ran, are the only place this traffic shows up at all. This is the practical follow-through on the log-check step in how to check AI crawler access, and the methodology behind identifying which bot is which is covered in auditing your access logs by user agent.
Where do the files actually live?
Log location depends on your stack. Common defaults:
- Apache:
/var/log/apache2/access.logor/var/log/httpd/access_log - Nginx:
/var/log/nginx/access.log - Shared hosting (cPanel): typically
/home/username/access-logs/ - Cloudflare / CDN-fronted sites: check the CDN's own logging product (e.g., Cloudflare Logpush). The origin server's logs alone may only show requests that made it past the edge, which is itself useful information if you're also checking for the CDN-layer blocking covered in the access-check piece. Cloudflare also now ships a native AI Crawl Control dashboard with a robots.txt-violation tracker (added October 2025) that lists exactly which crawlers are violating your directives, by crawler, path, and directive: a lower-effort alternative to a manual Logpush export for a quick check.
Beyond counting, what actually matters?
A raw hit count is the fast pass, not the analysis. It misses the response code, which specific paths actually get crawled, and the trend over time.
Start with the response code. grep -i "GPTBot" access.log | awk '{print $4, $7, $9}' prints timestamp, request path, and status code together. A string of 200s means the bot is actually receiving your content. A string of 403s or 429s means something is blocking or rate-limiting it, which a simple wc -l count would report as "traffic," not as "traffic that's being rejected."
Then check which paths get crawled. grep -i "GPTBot" access.log | awk '{print $7}' | sort | uniq -c | sort -rn | head -25 ranks the specific pages a bot actually requested. A bot that only ever hits your homepage and never your actual content pages is telling you something different than a bot with low overall volume: the first suggests a crawl-depth or internal-linking problem, not an access problem.
Last, look at the trend rather than the snapshot. grep "GPTBot" access.log | awk '{print $4}' | cut -d: -f2 | sort | uniq -c groups requests by hour, useful for seeing whether a bot's activity clusters or spreads out, but the more important trend is day-over-day and week-over-week, not hour-over-hour. One day's log is a small, noisy sample of a system with real variance built into it. The SparkToro/Gumshoe research already cited elsewhere on this site makes the same point about prompt-level AI visibility checks, and it applies just as directly here: a single day's crawl log is one roll, not a reading.
How do you build a repeatable check instead of a one-off audit?
A daily cron job that appends a summary to a running file gets most of the value without needing a dedicated tool:
grep "$(date +"%d/%b/%Y")" access.log | awk -F\" '{print $6}' | sort | uniq -c | sort -nr | head -10 >> bot_report.txt
That appends a daily ranked snapshot of the most active user agents to a single growing file. It's cheap to run, and it turns "did I check this once" into an actual trend you can look back on weekly or monthly. Whether weekly or monthly is the right cadence depends on your traffic volume; a high-traffic site can spot trend shifts weekly, while a lower-traffic one may need to accumulate a month of data before a change in bot behavior is distinguishable from normal noise.
What do good and bad look like?
A search/retrieval bot (OAI-SearchBot, Claude-SearchBot, PerplexityBot) showing consistent 200s across your actual content pages, at a stable or growing rate month over month, is the pattern that correlates with staying citable. A pattern of 403s or 429s where you expected 200s, or a bot that only ever reaches your homepage, is the pattern worth investigating before spending on anything downstream of access. Content extractability or earned off-site work don't matter if the crawler can't get past the front door in the first place.
Frequently asked questions
Why don't AI crawlers show up in Google Analytics or GA4?
Most AI crawlers don't execute JavaScript. They request the raw HTML and parse it directly, so the analytics tracking script embedded in your page never fires. Server access logs capture every request at the infrastructure level regardless of whether JavaScript ran, which is why they're the only reliable source for this traffic.
What should I actually check in server logs, beyond just counting hits?
Three things beyond a raw count: the response code returned to each bot (a 200 means it actually got your content; a 403 or 429 means something blocked or throttled it, which a hit count alone won't show), which specific paths get crawled versus ignored, and the trend over time rather than a single snapshot, since one day's log is a noisy sample of a system that varies day to day.
How often should I check AI crawler logs?
A recurring, scheduled check (weekly or monthly) beats a one-off audit, because crawl behavior genuinely varies over time and a single check is a snapshot, not a trend. A simple cron job that appends a daily summary to a running report file gets most of the value without needing a dedicated monitoring tool.
Do I need a paid tool to do this?
No: grep and awk against your raw access logs cover the entire practical workflow described here, and most hosting providers and CDNs give direct log access without extra cost. If you outgrow that, Screaming Frog's Log File Analyser is the standard named reference, free up to 1,000 log events, with built-in AI-bot presets and reverse-DNS bot verification. A paid tool becomes worth it once you want that kind of automated alerting or trend dashboard, not to do the basic analysis itself.
Sources referenced
- SparkToro / Gumshoe.ai AI recommendation consistency study (Jan 2026, ~2,961 prompt runs), cited here for the single-sample-vs-trend principle, applied to crawl logs rather than prompt runs
- HasData: The AI Crawler Block Index (Jul 2026, 10,894 domains)
- Screaming Frog: Monitor AI bots in the Log File Analyser
- Cloudflare changelog: Track robots.txt in AI Crawl Control (Oct 2025)
This piece is the practical follow-through on the log-check step in how to check AI crawler access, and pairs with auditing your access logs by user agent for verifying which bot is actually which. For whether the robots.txt rules those logs are checking against are actually being honored, see do AI crawlers respect robots.txt?
