Guide · 23 September 2026
Does your website exist to an AI crawler?
Fetch your own homepage the way a bot does, run
curl -s https://yoursite.com | sed 's/<[^>]*>//g' in a
terminal, and read what comes back. If it returns nothing but script tags,
every answer engine sees that same nothing, no matter how good the site looks in a
browser.
By Florian V. · Published 23 September 2026 · Two minutes. Every number below names its source and its year in the sentence.
Why it happens
Why does a site that looks fine in Chrome come back empty?
Because your browser runs your JavaScript and the crawlers do not. A modern site ships an almost empty HTML file plus a script that builds the page once it lands. You never see the gap.
01
The measurement
Vercel measured AI crawler traffic across its network and reported on 17 December 2024 that “none of the major AI crawlers currently render JavaScript”, naming GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot and PerplexityBot. They download your scripts and treat them as dead text.
02
Google is the exception
Googlebot renders JavaScript, and AI Overviews are built on Google's index. So a site can rank well and still be a blank page to ChatGPT, Perplexity and Claude. Search Console proves nothing about the rest.
03
Our own site failed it
We ran this on bva.marketing on 23 September 2026: 439 bytes, an empty body, one visible word, no h1, no meta description, no structured data. The official GPTBot/1.4 user agent got an HTTP 429 three tries out of three; a browser got a 200.
| Crawler | What it is for | Runs your JavaScript | If it cannot read you |
|---|---|---|---|
| Googlebot | Google Search and the index behind AI Overviews. | Yes | You lose organic search, and the AI Overviews built on it. |
| OAI-SearchBot | ChatGPT's search index. | No | OpenAI: sites opted out of it “will not be shown in ChatGPT search answers.” |
| GPTBot | Model training. A separate user agent from search. | No | Blocking it does not remove you from ChatGPT search. |
| ChatGPT-User | Live fetches, during someone's actual question. | No | A buyer is looking at you now, and the page is empty. |
| PerplexityBot | Perplexity's index; its docs ask you to allow it. | No | You do not appear in Perplexity results. |
| ClaudeBot | Anthropic's crawler; honours robots.txt per its docs. | No | Claude has nothing of yours to quote. |
The test
How do I run it, right now, on my own site?
Four commands. Replace yoursite.com with your domain, and read the output rather than the browser.
Read your homepage as plain text
Run curl -s https://yoursite.com | sed 's/<[^>]*>//g'.
It strips the tags and prints what is really in the file your server sent:
headline, offer, services, in readable English. Script paths and nothing else
means you have found the problem, and the rest of this page is secondary to
fixing it. No terminal? Use view-source, never the inspector.
Check that robots.txt is really a file
Run curl -I https://yoursite.com/robots.txt. You want a 200
and a content-type of text/plain. A single-page app often answers 200 with
text/html and hands back the same empty shell as every other URL, not
a robots file at all. That is what our own domain did on 23 September
2026.
Check that the sitemap exists
Run curl -s https://yoursite.com/sitemap.xml | head -20. You
want XML with your real URLs, named at the bottom of robots.txt. Five URLs
beats a missing file. Perplexity's documentation asks publishers to allow
PerplexityBot in robots.txt to appear in its results, so both files earn
their ten minutes.
Ask for the page as a bot, not as a browser
Run curl -I -A 'Mozilla/5.0 (compatible; GPTBot/1.4; +https://openai.com/gptbot)' https://yoursite.com/
and compare the status code with the one your browser gets. A 403 or a 429
means something in between, bot protection, a rate limiter, a firewall
rule, is turning the crawler away. Usually nobody chose it. Repeat with
OAI-SearchBot, PerplexityBot and ClaudeBot.
Reading the result
What does each outcome mean, and what do I fix first?
The text is missing
Your pages are built in the browser. Serve the text in the HTML instead: server rendering, static generation, or pre-rendering. Everything else depends on this. The acceptance test is step 01 returning readable prose.
The text is there, the labels are not
No title, no meta description, no h1, no structured data. Fixable the same afternoon: a descriptive title, one h1, a description saying what the page answers. Mark up only what a visitor can see, Google's policies treat hidden markup as a violation, not a shortcut.
A crawler gets a 403 or a 429
Check bot protection, rate limits and CDN rules first. Usually a switch somebody flipped to stop scrapers, now stopping the crawler that decides whether you get named.
After the fix
The page is readable. Why am I still not cited?
Because readable is the entry ticket, not the work. The one recipe tested under controlled conditions and published in a peer-reviewed venue is Aggarwal, Murahari, Rajpurohit, Kalyan, Narasimhan and Deshpande, “GEO: Generative Engine Optimization”, ACM SIGKDD (KDD 2024), on a benchmark of 10,000 queries.
The KDD 2024 paper's top methods, Cite Sources, Quotation Addition, Statistics Addition, achieved a relative improvement of 30 to 40% on the Position-Adjusted Word Count metric and 15 to 30% on the Subjective Impression metric. The same paper found keyword stuffing performed worse than the untouched original. So: answer in the first 40 to 60 words, date your sources, use numbers not adjectives. Narrative introductions get skipped.
What will not rescue you is structured data sold as a citation lever: Ahrefs tested 1,885 pages that added JSON-LD against matched controls in May 2026 and found no significant gain (+2.4% in Google AI Mode, +2.2% in ChatGPT, −4.6% in AI Overviews). Add it for entity clarity instead.
Read our GEO page for the full list of what the evidence supports and what it does not. We will not give you a date by which an engine re-crawls you: we do not control that.
Questions
Straight answers.
Does Googlebot read my site the same way ChatGPT does?
No. Googlebot renders JavaScript, so a client-rendered site can still rank in Google and appear in AI Overviews. Vercel's measurement of AI crawler traffic, published 17 December 2024, reported that none of the major AI crawlers render JavaScript, naming GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot and PerplexityBot. Passing in Google tells you nothing about the others.
Can I just use my browser's inspector to check what a bot sees?
No, and it is the trap that hides the problem. The inspector shows the DOM after your JavaScript has run, which is exactly what a non-rendering crawler never gets. Use view-source, or fetch the page with curl: both show the raw HTML your server sent.
Should I block AI crawlers in robots.txt?
Training and search are separate decisions with separate user agents, so you can allow one and refuse the other. If you want to be named in ChatGPT's answers, do not block its search crawler: OpenAI's documentation states that sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers. Perplexity likewise asks you to allow PerplexityBot.
Sources
Every claim above, and where it comes from.
Primary sources
- Vercel, “The rise of the AI crawler”, 17 December 2024. vercel.com/blog
- Aggarwal et al., “GEO: Generative Engine Optimization”, ACM SIGKDD (KDD 2024), Barcelona, August 2024. arxiv.org/abs/2311.09735
- OpenAI, bots documentation. developers.openai.com
- Perplexity, bots documentation. docs.perplexity.ai
- Google Search Central, AI features and structured data policies. developers.google.com/search
Measurements, caveat stated
- Ahrefs, 1,885 pages with matched controls, May 2026, Ahrefs sells SEO software.
- Our own measurement of bva.marketing, 23 September 2026: 439 bytes, body was a single empty div, the only word in the document was the title, no h1, no meta description, HTTP 429 to GPTBot/1.4.
Where a source sells a product in the field it measures, we say so in the same sentence. Our own result is reproducible with the four commands above, run them on us.
Free audit
Or we run all four checks on your site.
We fetch your pages as Googlebot, GPTBot, PerplexityBot and ClaudeBot, and send you the status code and readable word count each one gets, plus the three fixes worth doing first. In writing, within two business days. No call.