How do I test my robots.txt file?

Copy the file from yoursite.com/robots.txt, paste it into the tester, then enter a URL and a crawler. You see allowed or blocked, the group and rule that decided, and the result for 25 known crawlers. Nothing is fetched or stored.

What does “Disallow: /” mean in robots.txt?

It blocks the crawlers in that group from every URL on the site. Under User-agent: * it applies to every crawler without its own group. Use it on staging sites; it does not remove pages Google has already indexed.

How do I block GPTBot?

Add a group with User-agent: GPTBot and Disallow: /. OpenAI says this keeps your content out of its model training. It does not affect ChatGPT search, which uses OAI-SearchBot.

Anthropic’s crawler that collects web content which could help train its models. Anthropic says its bots honour robots.txt and Crawl-delay. Claude-SearchBot handles search and Claude-User fetches pages a user asks for, each with its own user agent.

Common Crawl’s crawler. It builds a free, open archive of the web, which AI models have been trained on: OpenAI’s GPT-3 paper drew 60% of its training mix from filtered Common Crawl data. CCBot checks robots.txt first and obeys Crawl-delay.

Does blocking Google-Extended affect Google Search?

No. Google says Google-Extended does not affect inclusion in Google Search and is not a ranking signal. It controls whether Google may use your content for Gemini training and grounding.

Does robots.txt stop a page from being indexed?

No. It stops crawling. Google says a blocked URL can still be indexed and shown without a snippet if other pages link to it. Use a noindex tag on a crawlable page, or password protection, to keep it out.

Does Google support crawl-delay?

No. Google reads only user-agent, allow, disallow and sitemap. Anthropic and Common Crawl say they honour Crawl-delay; Amazon says Amazonbot does not.

Which rule wins when Allow and Disallow both match?

The rule with the longer path wins. When both are the same length, Allow wins. So Allow: /folder beats Disallow: /folder, and Disallow: /*.htm beats Allow: /page on /page.htm.

Free SEO tool

Robots.txt tester & generator with AI crawler rules

Paste a robots.txt and test any URL against any bot, with the rule that decided shown. Or build a new file from presets that block AI training crawlers and keep AI search.

Updated October 27, 2026

Your robots.txt

robots.txt

Paste the whole file. Nothing is fetched: copy it from yoursite.com/robots.txt.

URL or path to test

A full URL or a path such as /blog/post?ref=x.

Crawler

A name from the list, any product token, or a full User-Agent header.

Crawler list verified October 27, 2026

Runs in your browser. Nothing you type is sent or stored.

Googlebot

Allowed

Googlebot may fetch /app/pricing.

Deciding rule: Allow: /app/pricing$line 5
The longest matching rule wins; Allow wins a tie.
Group it obeys: User-agent: * (no group names this crawler)
Other matching rules: Disallow: /app/ line 3

Compared as /app/pricing

This path for every known crawler

Same file, same path, 25 crawlers. “Follows robots.txt” is what each vendor says about its own agent.

Crawler	Result	Rule	Follows robots.txt
GPTBotOpenAI · AI training	Blocked	Disallow: /	Yes
ClaudeBotAnthropic · AI training	Allowed	Allow: /app/pricing$	Yes
Google-ExtendedGoogle · AI training	Allowed	no rule	Yes
Applebot-ExtendedApple · AI training	Allowed	Allow: /app/pricing$	Yes
CCBotCommon Crawl · AI training	Blocked	Disallow: /	Yes
meta-externalagentMeta · AI training	Allowed	Allow: /app/pricing$	Yes
AmazonbotAmazon · AI training	Allowed	Allow: /app/pricing$	Yes
MistralAI-TrainingMistral AI · AI training	Allowed	Allow: /app/pricing$	Yes
GooglebotGoogle · Search	Allowed	Allow: /app/pricing$	Yes
BingbotMicrosoft · Search	Allowed	Allow: /app/pricing$	Yes
ApplebotApple · Search	Allowed	Allow: /app/pricing$	Yes
OAI-SearchBotOpenAI · Search	Allowed	Allow: /app/pricing$	Yes
Claude-SearchBotAnthropic · Search	Allowed	Allow: /app/pricing$	Yes
PerplexityBotPerplexity · Search	Allowed	Allow: /app/pricing$	Yes
meta-webindexerMeta · Search	Allowed	Allow: /app/pricing$	Yes
Amzn-SearchBotAmazon · Search	Allowed	Allow: /app/pricing$	Yes
DuckAssistBotDuckDuckGo · Search	Allowed	Allow: /app/pricing$	Yes
MistralAI-IndexMistral AI · Search	Allowed	Allow: /app/pricing$	Yes
ChatGPT-UserOpenAI · User-triggered fetch	Allowed	Allow: /app/pricing$	Not always
Claude-UserAnthropic · User-triggered fetch	Allowed	Allow: /app/pricing$	Yes
Perplexity-UserPerplexity · User-triggered fetch	Allowed	Allow: /app/pricing$	Generally no
Google-AgentGoogle · User-triggered fetch	Allowed	Allow: /app/pricing$	Generally no
meta-externalfetcherMeta · User-triggered fetch	Allowed	Allow: /app/pricing$	Not always
Amzn-UserAmazon · User-triggered fetch	Allowed	Allow: /app/pricing$	Not always
MistralAI-UserMistral AI · User-triggered fetch	Allowed	Allow: /app/pricing$	Yes

Checks

Note · Line 11Google-Extended only controls how crawled content is used for AI. Blocking it does not remove your pages from that company’s search.

Quick answer

How does a robots.txt tester decide if a URL is blocked?

It finds the group whose User-agent names the crawler, or the * group when none does. Among that group’s rules matching the path, the longest rule wins and Allow wins a tie. With Disallow: /blog/ and Allow: /blog/public/, /blog/public/post is allowed (13 characters beat 6) and /blog/draft is blocked.

How to use the robots.txt tester and generator

Paste your file
Open yoursite.com/robots.txt, copy everything and paste it into Test a URL. The checks list typos, ignored lines and risky rules.
Pick a URL and a crawler
Enter a full URL or a path, and a crawler such as Googlebot, GPTBot or ClaudeBot. You can also paste a full User-Agent header from your logs.
Read the verdict
See allowed or blocked, the group the crawler obeys and the rule that decided, then the same path for all 25 known crawlers.
Generate a new file
Switch to Generate, start from a preset, pick crawlers to block, add paths and sitemaps, then copy or download robots.txt and test it.

How the matching works

A crawler reads the groups in the file and obeys the one whose User-agent equals its product token, ignoring case. Two groups for the same crawler merge into one. If no group names it, it obeys the * group; if there is no * group, nothing is blocked. A named group and the * group never combine, so a GPTBot group replaces the * rules for GPTBot.

Inside the group, every Allow and Disallow rule is compared with the URL path and query. A rule matches when the path starts with it; * stands for any run of characters and a $ at the end means the path must end there. The longest matching rule wins, measured on the rule as written, and Allow wins when the lengths are equal. No match means allowed.

Paths compare in percent-encoded form, so /café and /caf%C3%A9 are the same rule, and matching is case-sensitive. The tester follows RFC 9309, the robots.txt standard published in 2022, and Google’s documented reading of it, including Google’s 500 KiB size limit.

Formula

Group = groups naming the crawler, else the * groups, else none
Winner = matching rule with the longest path; Allow wins a tie
No matching rule → allowed

Worked example

File: User-agent: * / Disallow: /app/ / Allow: /app/pricing$
URL: /app/pricing, crawler: Googlebot.
No Googlebot group, so Googlebot obeys User-agent: *.
Matching rules: Disallow: /app/ (5 characters) and Allow: /app/pricing$ (13 characters).
13 beats 5, so Allow: /app/pricing$ decides: allowed. /app/pricing/annual is blocked, because $ ends the Allow rule.

AI crawler user agents, October 2026

What each vendor says its crawler is for and whether it follows robots.txt, from the vendor’s own documentation, checked 2026-10-27.

User agent	Company	Purpose	Follows robots.txt	What the vendor says	Source
GPTBot	OpenAI	AI training	Yes	Crawls content that may be used to train OpenAI’s foundation models. Disallowing it opts out of training only.	developers.openai.com
ClaudeBot	Anthropic	AI training	Yes	Collects content that could help train Anthropic’s models. Anthropic also honours Crawl-delay.	support.claude.com
Google-Extended	Google	AI training	Yes	Controls use for Gemini training and grounding. No effect on Google Search inclusion or ranking.	developers.google.com
Applebot-Extended	Apple	AI training	Yes	Controls use for training Apple’s foundation models. Does not crawl; pages stay eligible for Apple search.	support.apple.com
CCBot	Common Crawl	AI training	Yes	Builds Common Crawl’s open web archive, which AI models have been trained on. Obeys Crawl-delay.	commoncrawl.org
meta-externalagent	Meta	AI training	Yes	Crawls to train Meta’s foundation AI models or improve products by indexing content.	developers.facebook.com
Amazonbot	Amazon	AI training	Yes	Improves Amazon products and may be used to train Amazon AI models. Ignores Crawl-delay.	developer.amazon.com
MistralAI-Training	Mistral AI	AI training	Yes	Crawls content to build datasets for training Mistral’s generative models.	docs.mistral.ai
Googlebot	Google	Search	Yes	Google Search, including Discover, Images and News. Ignores Crawl-delay.	developers.google.com
Bingbot	Microsoft	Search	Yes	Bing’s standard crawler. Robots.txt controls how it crawls your site.	www.bing.com
Applebot	Apple	Search	Yes	Search in Spotlight, Siri and Safari. Follows your Googlebot rules when no Applebot group exists.	support.apple.com
OAI-SearchBot	OpenAI	Search	Yes	Surfaces sites in ChatGPT search. Blocked sites are left out of its answers but can appear as navigational links.	developers.openai.com
Claude-SearchBot	Anthropic	Search	Yes	Analyses content to improve the relevance and accuracy of Claude’s search results.	support.claude.com
PerplexityBot	Perplexity	Search	Yes	Surfaces and links sites in Perplexity search. Not used to crawl for AI foundation models.	docs.perplexity.ai
meta-webindexer	Meta	Search	Yes	Improves Meta AI search results and helps Meta AI cite and link to content.	developers.facebook.com
Amzn-SearchBot	Amazon	Search	Yes	Search in Amazon products such as Alexa. Not used for generative AI training.	developer.amazon.com
DuckAssistBot	DuckDuckGo	Search	Yes	Fetches pages in real time for DuckDuckGo’s AI-assisted answers. Not used to train AI models.	duckduckgo.com
MistralAI-Index	Mistral AI	Search	Yes	Indexes content for Mistral search. Not used for generative AI training.	docs.mistral.ai
ChatGPT-User	OpenAI	User-triggered fetch	Not always	Visits a page when a ChatGPT user asks. OpenAI says robots.txt rules may not apply.	developers.openai.com
Claude-User	Anthropic	User-triggered fetch	Yes	Fetches a page when a Claude user asks a question that needs it.	support.claude.com
Perplexity-User	Perplexity	User-triggered fetch	Generally no	Visits a page to answer a user’s question. Perplexity says it generally ignores robots.txt.	docs.perplexity.ai
Google-Agent	Google	User-triggered fetch	Generally no	Browses and acts on the web for a user. Google says user-triggered fetchers generally ignore robots.txt.	developers.google.com
meta-externalfetcher	Meta	User-triggered fetch	Not always	Fetches links at a user’s request. Meta says it may bypass robots.txt.	developers.facebook.com
Amzn-User	Amazon	User-triggered fetch	Not always	Fetches live information for a user, such as an Alexa answer. May not follow all robots.txt rules.	developer.amazon.com
MistralAI-User	Mistral AI	User-triggered fetch	Yes	Visits a page when a user asks Vibe, Mistral’s assistant, a question. Its robots.txt tag governs those requests.	docs.mistral.ai

Each row names the vendor page it comes from; the links are under Sources. Bytespider is missing because no ByteDance page documenting it could be reached. A user agent string can be faked: OpenAI, Perplexity, Bing and Common Crawl publish IP ranges or lookups to verify real visits.

Block AI training without leaving AI search

Training, search and user-triggered fetches use different user agents, and each vendor treats them separately. OpenAI says disallowing GPTBot keeps content out of training, while OAI-SearchBot decides whether you appear in ChatGPT search. Google-Extended and Applebot-Extended never crawl: they only control AI use of what Googlebot and Applebot fetch, and blocking them keeps you in Google and Apple search. User-triggered fetchers such as ChatGPT-User and Perplexity-User may ignore robots.txt, because a person asked for the page.

Robots.txt disallow all, and when not to use it

User-agent: * with Disallow: / asks every crawler to stay out of the whole site. It suits a staging site, not a live one. It does not remove pages already in Google: Google says a disallowed URL can still be indexed and shown without a snippet if other pages link to it. To keep a page out of results, let it be crawled and add noindex, or put it behind a password.

Common robots.txt mistakes the checker catches

Mistake	What happens	Fix
Disallow: / under User-agent: *	Every crawler without its own group, Googlebot included, skips the whole site.	Keep it for staging; on a live site block only the paths you mean.
A path without a leading slash	Disallow: admin never matches, because every path starts with /.	Write Disallow: /admin.
Rules above the first User-agent line	Crawlers ignore them.	Put a User-agent line first.
User-agent: * on its own line above User-agent: GPTBot	Both lines form one group, so the GPTBot rules apply to everyone.	Give the * group its own rule, such as Allow: /.
Crawl-delay for Googlebot	Google ignores the field.	Leave it out for Google; some other crawlers honour it.
Sitemap: /sitemap.xml	Google asks for a full URL.	Write Sitemap: https://example.com/sitemap.xml.
Misspelled fields such as Dissallow	Crawlers that follow RFC 9309 skip the line.	Spell it Disallow.

Robots.txt questions

How do I test my robots.txt file?: Copy the file from yoursite.com/robots.txt, paste it into the tester, then enter a URL and a crawler. You see allowed or blocked, the group and rule that decided, and the result for 25 known crawlers. Nothing is fetched or stored.
What does “Disallow: /” mean in robots.txt?: It blocks the crawlers in that group from every URL on the site. Under User-agent: * it applies to every crawler without its own group. Use it on staging sites; it does not remove pages Google has already indexed.
How do I block GPTBot?: Add a group with User-agent: GPTBot and Disallow: /. OpenAI says this keeps your content out of its model training. It does not affect ChatGPT search, which uses OAI-SearchBot.
What is ClaudeBot?: Anthropic’s crawler that collects web content which could help train its models. Anthropic says its bots honour robots.txt and Crawl-delay. Claude-SearchBot handles search and Claude-User fetches pages a user asks for, each with its own user agent.
What is CCBot?: Common Crawl’s crawler. It builds a free, open archive of the web, which AI models have been trained on: OpenAI’s GPT-3 paper drew 60% of its training mix from filtered Common Crawl data. CCBot checks robots.txt first and obeys Crawl-delay.
Does blocking Google-Extended affect Google Search?: No. Google says Google-Extended does not affect inclusion in Google Search and is not a ranking signal. It controls whether Google may use your content for Gemini training and grounding.
Does robots.txt stop a page from being indexed?: No. It stops crawling. Google says a blocked URL can still be indexed and shown without a snippet if other pages link to it. Use a noindex tag on a crawlable page, or password protection, to keep it out.
Does Google support crawl-delay?: No. Google reads only user-agent, allow, disallow and sitemap. Anthropic and Common Crawl say they honour Crawl-delay; Amazon says Amazonbot does not.
Which rule wins when Allow and Disallow both match?: The rule with the longer path wins. When both are the same length, Allow wins. So Allow: /folder beats Disallow: /folder, and Disallow: /*.htm beats Allow: /page on /page.htm.

Cite this tool

Free to link and quote. Use this reference:

SaaSCity, "Robots.txt Tester & Generator", October 27, 2026. https://saascity.io/free-tools/robots-txt-generator