AI Crawler Traffic: a 17-Day Measurement
2026-07-15 to 2026-07-31 (17 days)
This page publishes primary data collected by TYS Digital Performance from its own server logs. The measurement period runs from 2026-07-15 to 2026-07-31 and covers 17 days. During that period the server received 326,764 requests in total. Of those, 15,278 came from AI crawlers and 11,585 from classic search engine crawlers. AI crawlers therefore made 31.9 percent more requests than search engine crawlers.
AI crawler requests by provider
| Provider | Crawler | Requests | Share |
|---|---|---|---|
| Meta | meta-externalagent | 4,673 | 30.6 % |
| Anthropic | ClaudeBot, anthropic-ai | 4,280 | 28 % |
| OpenAI | OAI-SearchBot, GPTBot, ChatGPT-User | 1,949 | 12.8 % |
| ByteDance | Bytespider | 1,745 | 11.4 % |
| Amazon | Amazonbot | 1,404 | 9.2 % |
| Perplexity | PerplexityBot | 896 | 5.9 % |
| Common Crawl | CCBot | 317 | 2.1 % |
| Apple | Applebot-Extended | 14 | 0.1 % |
| Total | 15,278 | 100 % |
For comparison: classic search engine crawlers
| Crawler | Requests |
|---|---|
| Googlebot | 7,353 |
| Applebot | 2,720 |
| bingbot | 983 |
| YandexBot | 508 |
| DuckDuckBot | 21 |
| Total | 11,585 |
HTTP status codes returned to AI crawlers
| Status code | Requests |
|---|---|
| 200 | 13,787 |
| 301 | 1,003 |
| 404 | 320 |
| 308 | 104 |
| 307 | 34 |
| 206 | 21 |
| 502 | 5 |
| 503 | 2 |
| 400 | 2 |
Method
- The data source is the shared nginx access logs of the domains tysd.de and tys.net.tr. The raw logs are kept on the server and retained for 365 days.
- Crawlers were classified by the identifier in the user-agent header. Counted as AI crawlers: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, anthropic-ai, Google-Extended, PerplexityBot, Amazonbot, Bytespider, CCBot, meta-externalagent, Applebot-Extended. Counted as classic search crawlers: Googlebot, bingbot, Applebot, YandexBot, DuckDuckBot.
- Each request was counted in one category only. Where a user-agent matched both lists it was counted as an AI crawler, so the search engine figures are a lower bound.
- 412 lines (0.13 percent of the total) could not be parsed because of a format mismatch and were excluded from every total.
- The figures are reproducible: the same log files and the same classification list produce the same result.
Limits of this measurement
- This is data from a single property. It was measured on an agency's own website and cannot be generalised to the sector.
- Crawler identity rests on the user-agent header. That header can be spoofed and was not verified here by reverse DNS.
- What was measured is the number of requests. A crawler fetching a page does not show that the page is cited in an answer.
- The window covers 17 days. Seasonality, one-off crawl waves and changes to the site may have affected the result.
- The measurement period coincides with structural changes to the site; part of the 1,003 redirects and the 320 not-found responses stems from those changes.
What this measurement does not show
These figures measure crawl intensity and are not a visibility promise. More crawler traffic does not raise citations, rankings or visitor numbers. No such relationship was measured with this data and none is claimed here.