ChatGPT Now Crawls 3.6x More Than Googlebot: What 24M Requests Reveal
An analysis of 24.4 million requests reveals: AI crawlers now make 3.6x more requests than traditional search engine crawlers. ChatGPT-User leads the list, ahead of Googlebot, Amazonbot, and Bingbot combined. What this means for your SEO strategy.
Most website operators assume Googlebot is the dominant crawler on their site. That assumption is now outdated. An analysis of 24.4 million proxy requests across 69 websites (January to March 2026) reveals a new reality: OpenAI's ChatGPT-User crawler made 3.6x more requests than Googlebot.
This article summarizes the key findings from the original article on Search Engine Journal (Author: Kyle Duck / Alli AI, April 7, 2026).
1. AI Crawlers Dominate: The New Numbers
Top 10 crawlers by request volume:
| Rank | Crawler | Requests | Category |
|---|---|---|---|
| 1 | ChatGPT-User (OpenAI) | 133,361 | AI Search |
| 2 | Googlebot | 37,426 | Traditional Search |
| 3 | Amazonbot | 35,728 | AI / E-Commerce |
| 4 | Bingbot | 18,280 | Traditional Search |
| 5 | ClaudeBot (Anthropic) | 13,918 | AI Search |
| 6 | MetaBot | 10,756 | Social |
| 7 | GPTBot (OpenAI) | 8,864 | AI Training |
| 8 | Applebot | 6,794 | AI Search |
| 9 | Bytespider (ByteDance) | 6,644 | AI Training |
| 10 | PerplexityBot | 5,731 | AI Search |
ChatGPT-User alone made more requests than Googlebot, Amazonbot, and Bingbot combined. Total AI-related crawlers: 213,477 requests vs. 59,353 for traditional search crawlers.
2. OpenAI Uses Two Different Crawlers
A common mistake: many sites treat OpenAI as a single crawler. In reality, there are two with completely different purposes:
- ChatGPT-User is the retrieval crawler. It fetches pages in real time when users ask ChatGPT questions requiring current web information. It determines whether your content appears in ChatGPT's answers.
- GPTBot is the training crawler. It collects data to improve OpenAI's models. Training data teaches the models about your brand, products, and expertise.
Combined, OpenAI's crawlers made 142,225 requests: 3.8x Googlebot's volume.
Important for robots.txt: The directives are separate. If you block GPTBot but allow ChatGPT-User (or vice versa), understand the distinct consequences. The recommendation: allow both crawlers, since blocking the training crawler means AI models learn less about your business, reducing your chances of being cited in AI responses long-term.
3. AI Crawlers Are Faster and More Reliable
| Crawler | Avg Response Time | Success Rate (200) |
|---|---|---|
| PerplexityBot | 8ms | 100% |
| ChatGPT-User | 11ms | 99.99% |
| GPTBot | 12ms | 99.9% |
| ClaudeBot | 21ms | 99.9% |
| Bingbot | 42ms | 98.4% |
| Googlebot | 84ms | 96.3% |
Why? AI retrieval crawlers fetch specific pages in response to user queries. They know what they want, grab it, and leave. Googlebot, by contrast, exhaustively crawls entire site architectures, including stale URLs from sitemaps and its legacy index.
But: the sheer volume means aggregate server load is substantial. ChatGPT-User at 11ms x 133,361 requests is a real infrastructure cost.
4. Googlebot Sees a Worse Version of Your Site
Googlebot's 96.3% success rate vs. near-perfect rates for AI crawlers reveals a structural difference. Googlebot received 624 blocked responses (403) and 480 not-found errors (404), accounting for 3% of its requests.
The reason: Googlebot maintains a massive legacy index and routinely re-requests URLs that have been deleted or restructured. AI crawlers don't carry that baggage. They fetch specific pages that are currently relevant and linked.
Industry Reports Confirm the Trend
The findings align with broader industry data:
- Cloudflare's 2025 analysis reported ChatGPT-User requests surging 2,825% YoY.
- Akamai identified OpenAI as the single largest AI bot operator at 42.4% of all AI bot requests.
- Vercel's analysis confirmed that none of the major AI crawlers currently render JavaScript.
What You Should Do Now
- Audit robots.txt for AI crawlers: Set up explicit directives for ChatGPT-User, GPTBot, ClaudeBot, Amazonbot, PerplexityBot, Applebot, Bytespider, CCBot, and Google-Extended.
- Clean up stale URLs in Google Search Console: Googlebot's 3% error rate mostly comes from legacy URLs. Set up proper redirects and submit updated sitemaps.
- Treat AI crawler accessibility as a distinct SEO channel: Appearing in ChatGPT's answers, Perplexity's results, and Claude's responses is a standalone visibility channel. JavaScript-heavy frameworks without server-side rendering are invisible to AI crawlers.
- Plan for volume, not just individual requests: AI crawlers send light, fast requests, but many of them. Make sure your hosting and CDN can handle the volume.
Source: Search Engine Journal: "ChatGPT Now Crawls 3.6x More Than Googlebot: What 24M Requests Reveal", Kyle Duck / Alli AI, April 7, 2026.
