AI Crawler Statistics Shopify Merchants Should Know

AI crawler traffic across the open web grew 18% between May 2024 and May 2025, and GPTBot’s own request volume rose 305% that year, according to Cloudflare. Almost none of that shows up in a Shopify store’s own analytics, since it never touches checkout. Here are the sourced numbers behind it, each dated and pulled from primary research.

TL;DR

Number	What it means	What to do about it
AI and search crawler traffic up 18% in one year (Cloudflare)	Crawling load from these bots is rising, not flattening	Check your own server logs or Cloudflare dashboard for crawler share instead of assuming it is zero
GPTBot’s own requests up 305% in the same year (Cloudflare)	The most cited AI crawler is still climbing fast	Decide on purpose whether GPTBot should read your store, rather than leaving robots.txt silent by accident
Only 14% of major sites’ robots.txt files mention an AI bot at all (Cloudflare)	Most sites, Shopify stores included, have made no explicit choice either way	Add a clear allow or disallow line instead of staying silent
GPTBot is blocked by 5.89% of ~140 million sites, more than any other AI bot (Ahrefs)	Blocking is still rare, but ClaudeBot’s block rate is growing the fastest	Review any block you already have against what the bot is actually for
AI-referred US retail traffic up 1,219% since October 2024, converting 60% higher (Adobe)	The AI traffic that does arrive already buys at a better rate than other traffic	Track AI referrals as their own line, not folded into “other”

How Fast Is AI Crawler Traffic Actually Growing?

Cloudflare Radar tracked a cohort of more than 30 AI and search crawlers across its own network and found their combined traffic up 18% from May 2024 to May 2025, comparing full calendar months, and up 48% once newly added Cloudflare customers are counted in. Traffic actually peaked in April 2025, 32% above the May 2024 baseline, before easing slightly. Within that cohort, the individual crawlers moved at very different speeds. GPTBot’s share of all tracked crawler traffic rose from 2.2% to 7.7%, a 305% jump in raw requests that pushed it from the ninth-busiest crawler to the third. OpenAI’s own crawler documentation distinguishes GPTBot, which gathers training data, from ChatGPT-User, the crawler behind a live user’s own ChatGPT browsing; the latter grew 2,825% in requests over the same year, and PerplexityBot grew 157,490% from a base near zero. Not every AI crawler gained ground: ClaudeBot’s share fell from 11.7% to 5.4%, a 46% drop in requests. Googlebot still dwarfs all of them, holding 30% of this cohort’s traffic in May 2024 and 50% a year later, up 96% in raw requests, which is worth remembering before assuming “AI crawler” means anything other than Google crawling more.

Cloudflare Radar, crawler-traffic cohort of 30+ bots, published 1 Jul 2026.

Stat	Source	Date
AI and search crawler traffic up 18% from May 2024 to May 2025 across Cloudflare’s network	Cloudflare Radar	1 Jul 2026
Googlebot’s share of that cohort rose from 30% to 50%, up 96% in raw requests	Cloudflare Radar	1 Jul 2026
GPTBot’s share rose from 2.2% to 7.7%, up 305% in raw requests, moving from rank 9 to rank 3	Cloudflare Radar	1 Jul 2026
ChatGPT-User requests grew 2,825% in the same year	Cloudflare Radar	1 Jul 2026
PerplexityBot requests grew 157,490% from a near-zero base	Cloudflare Radar	1 Jul 2026
ClaudeBot’s share fell from 11.7% to 5.4%, down 46% in raw requests	Cloudflare Radar	1 Jul 2026

Cloudflare Radar, raw request growth May 2024 to May 2025, published 1 Jul 2026.

A crawler reaching a page is the first step before anything else in search or AI answers can happen to it; how Google crawls a Shopify store walks through that mechanism in diagram form, and our own guide to getting a Shopify store indexed covers what has to be true before any of this traffic can convert into a listing at all.

How Many Sites Actually Block These Crawlers?

Most do not, and most have not even made a decision about it. Ahrefs checked robots.txt across roughly 140 million websites and found GPTBot the single most-blocked AI bot of the 21 it tracks, disallowed by 5.89% of sites, with Common Crawl’s CCBot close behind at 5.85%. ClaudeBot’s block rate grew faster than any other tracked bot’s over the prior year, up 32.67%, even though it still sits well behind GPTBot in absolute terms. Separately, Cloudflare’s own check of the top 10,000 domains found only 14% of the ones with a reachable robots.txt file (546 of 3,816 checked) said anything about an AI bot at all, allow or disallow; the rest are silent, which by default leaves those bots free to crawl. Among the sites that do take a position, GPTBot is both the most blocked (312 domains) and the most explicitly allowed (61 domains), which says less about GPTBot itself than about how split site owners still are on the question.

Ahrefs, ~140 million sites, published 21 May 2026; Cloudflare Radar, top 10,000 domains, published 1 Jul 2026.

Stat	Source	Date
GPTBot is the most-blocked of 21 tracked AI bots, disallowed by 5.89% of ~140 million sites	Ahrefs	21 May 2026
CCBot (Common Crawl) is a close second, blocked by 5.85% of sites	Ahrefs	21 May 2026
ClaudeBot’s block rate grew 32.67% over the prior year, the fastest of any tracked bot	Ahrefs	21 May 2026
Only 14% of the top 10,000 domains’ robots.txt files (546 of 3,816) mention an AI bot at all	Cloudflare Radar	1 Jul 2026
GPTBot is also the most explicitly allowed AI bot among those same domains, named in 61 allow rules	Cloudflare Radar	1 Jul 2026

This is a different number from our own crawl of 148 live Shopify stores, which found almost no Shopify-specific blocking at all: Shopify merchants, as a group, look closer to the silent 86% here than to the sites actively weighing a block either way. StoreSEO’s robots.txt guidance exists for the merchants who want to move out of that silent majority on purpose: our full Shopify robots.txt guide covers exactly which lines Shopify lets you edit and which bot names to use, and the same product documents what to check first when a store is not showing up in Google at all, the same diagnostic path whether the crawler in question is Googlebot or an AI bot. If you want to see what your own store currently allows before changing anything, AIScan’s free agent-readiness checker reads a live site’s robots.txt and reports which named crawlers it currently welcomes or shuts out.

Does Any Of This Traffic Turn Into Sales?

For the traffic that does arrive, yes, and the margin is getting wider. Adobe Analytics, working from more than a trillion visits to US retail sites, found AI-referred traffic up 1,219% since October 2024 and up 62% year over year in July 2026 alone. Shoppers arriving from an AI referral convert at a rate 60% higher than non-AI traffic, the eleventh straight month AI traffic has outperformed, and they generate 53% more revenue per visit. The same visitors engage more once they land: 14% higher engagement, 59% more time on site, 33% less bounce, and 28% more add-to-cart activity than other shoppers. Readiness has not caught up with the demand, though. Adobe’s broader retail cohort averaged only 61% of homepage content readable by an LLM as of July 2026, which means 39% of a typical retailer’s homepage is effectively invisible to the same crawlers driving this traffic, with apparel (76% readable) well ahead of grocery (59%).

Adobe Analytics via Digital Commerce 360, July 2026 data, published 19 Aug 2026.

Stat	Source	Date
AI-referred traffic to US retail sites up 1,219% since October 2024	Adobe Analytics, via Digital Commerce 360	19 Aug 2026
That traffic was up 62% year over year in July 2026 alone	Adobe Analytics, via Digital Commerce 360	19 Aug 2026
AI-referred visits convert 60% higher than non-AI traffic, the 11th straight month of outperformance	Adobe Analytics, via Digital Commerce 360	19 Aug 2026
AI-referred shoppers generate 53% more revenue per visit	Adobe Analytics, via Digital Commerce 360	19 Aug 2026
AI-referred visitors spend 59% more time on site and bounce 33% less	Adobe Analytics, via Digital Commerce 360	19 Aug 2026
39% of Adobe’s retail cohort’s homepage content is not readable by an LLM, varying 24% to 41% by category	Adobe Analytics, via Digital Commerce 360	19 Aug 2026

Against that, it is worth keeping a sense of scale. Ahrefs’ own traffic-mix study of roughly 35,000 sites in its Analytics product found AI referrals at just 0.1% of total referral traffic, far behind search (43.8%) and direct (42.3%). Both numbers are true at once: the growth rate is real and the current share of total traffic is still small, which is exactly why the trend line matters more than any single month’s total right now.

Which Of These Numbers Are Already Out Of Date?

Four cohorts, four methods, four dates, none of them meant to be averaged together.

These four sources measure different cohorts over different windows, and their numbers were never going to agree with each other down to the decimal. Cloudflare’s crawler-share figures come from its own network’s traffic and change every time it refreshes Radar; a report citing a different month will show different percentages for the same bots, not a contradiction so much as a different snapshot. Ahrefs’ block-rate study is a single pass across its own web index, dated to the day it ran, and a bot’s rank can move the next time anyone re-runs it, which is exactly what happened to ClaudeBot between the two Cloudflare snapshots a few months apart. Adobe’s retail figures are the fastest-moving of the four, reported monthly, so the “1,219% since October 2024” figure above gets a new base for comparison every time Adobe or Digital Commerce 360 publishes again. None of the four is more “correct” than another; each is correct for the cohort, method and date stated next to it, which is the point of carrying that date on every row above rather than averaging them into one number. This page gets rechecked against fresh Cloudflare, Ahrefs and Adobe data every quarter.

Frequently Asked Questions (FAQs)

1. Which AI crawler should a Shopify merchant actually pay attention to?

Googlebot first, since it still carries roughly half of all tracked crawler traffic and directly affects ordinary search visibility. After that, GPTBot is the single fastest-growing named AI crawler by raw request volume, which makes it the one most worth checking in your own server logs or Cloudflare dashboard before deciding whether to allow or block it.

2. Does blocking GPTBot or ClaudeBot actually stop them from using your content?

Only if the bot respects robots.txt, which is a voluntary standard, not an enforced one. A disallow line tells a well-behaved crawler not to request the page; it does not prevent a crawler that ignores the file, and it has no effect at all on content the bot already fetched before the rule was added.

3. Why do different reports give different percentages for the same AI bot?

Because they measure different things. Cloudflare’s numbers come from its own network traffic over a specific month, Ahrefs’ come from a single crawl of robots.txt files across its own web index, and Adobe’s come from retail-site visit data. Different cohorts, different methods and different dates will never line up exactly, which is why every stat above carries its own source and date instead of being blended into one figure.

4. Is AI referral traffic big enough for a small Shopify store to care about yet?

The dollar share is still small industry-wide, with one study putting AI referrals at just 0.1% of total traffic across the sites it measured. The growth rate is the reason to act now rather than later: traffic up over 1,000% since late 2024 and converting 60% better than other traffic is worth tracking as its own line before it is a meaningful share of revenue, not after.

5. How do I find out what my own Shopify store currently allows or blocks?

Open your store’s robots.txt directly in a browser at yourdomain.com/robots.txt and look for lines naming GPTBot, ClaudeBot, PerplexityBot or similar. A free scan from a tool like AIScan’s agent-readiness checker reports the same thing without reading the raw file by hand, and StoreSEO’s own robots.txt guide explains which of those lines a Shopify merchant can actually edit.

6. Will these numbers still be accurate next year?

AI crawler share reshuffles every time Cloudflare publishes a new snapshot, and Adobe’s retail figures update monthly, so several of the numbers above will have already moved. The dates attached to each row exist so a reader checking back later can judge which ones are likely stale.

Growth this fast does not mean every number here will hold for long, but the direction across all four sources agrees: more AI crawlers, more of them reaching Shopify-style sites by default, and more of the traffic that does arrive turning into a sale. Make sure your own robots.txt reflects a decision rather than silence, then try StoreSEO on the Shopify App Store.