What does your robots.txt allow?
Test any path against GPTBot, ClaudeBot, PerplexityBot, Googlebot and 17 more bots, and see the exact line that decides each one.
Free test. No sign-up. Made by Sunny Patel.
Example A check I ran on 19 November 2026 for sunnypatel.co.uk
Every bot I tested is allowed to fetch /.
This is what the file said when I read it. I match rules the way RFC 9309 and Google describe, and each crawler makes its own call.
Checked on 19 November 2026 at 18:53 GMT.
- Answer
- Read
HTTP 200 - Size
206 bytes- Content type
text/plain; charset=utf-8

The line that decides for GPTBot at /
Line 2Allow: /
Allowedin the * group
Allow: / (line 2) in the * group matches / and no Disallow rule does, so GPTBot is allowed.
Your robots.txt, line by line
9 lines. Marked lines carry a word and an icon: Decides is the line a bot's answer comes from.
- 1
User-Agent: * - 2
Allow: / - 3
Disallow: /keystatic/ - 4
Disallow: /api/ - 5
Disallow: /for/ - 6
- 7
Host: https://sunnypatel.co.uk - 8
Sitemap: https://sunnypatel.co.uk/sitemap.xml - 9
EntityMap: https://sunnypatel.co.uk/entitymap.json
Each bot, for /
21 bots tested: 21 allowed, 0 disallowed, 0 with no rule. A bot with no rule may fetch the path.
-
GPTBotOpenAI, AI training
Allowed
Group
*every botAllow: /Allow: / (line 2) in the * group matches / and no Disallow rule does, so GPTBot is allowed.
-
ClaudeBotAnthropic, AI training
Allowed
Group
*every botAllow: /Allow: / (line 2) in the * group matches / and no Disallow rule does, so ClaudeBot is allowed.
-
PerplexityBotPerplexity, AI search answers
Allowed
Group
*every botAllow: /Allow: / (line 2) in the * group matches / and no Disallow rule does, so PerplexityBot is allowed.
-
GooglebotGoogle, search results
Allowed
Group
*every botAllow: /Allow: / (line 2) in the * group matches / and no Disallow rule does, so Googlebot is allowed.
Every other bot I tested (17)
-
Google-ExtendedGoogle, AI training
Allowed
Group
*every botAllow: /Allow: / (line 2) in the * group matches / and no Disallow rule does, so Google-Extended is allowed.
-
GoogleOtherGoogle, other uses
Allowed
Group
*every botAllow: /Allow: / (line 2) in the * group matches / and no Disallow rule does, so GoogleOther is allowed.
-
BingbotMicrosoft, search results
Allowed
Group
*every botAllow: /Allow: / (line 2) in the * group matches / and no Disallow rule does, so Bingbot is allowed.
-
OAI-SearchBotOpenAI, AI search answers
Allowed
Group
*every botAllow: /Allow: / (line 2) in the * group matches / and no Disallow rule does, so OAI-SearchBot is allowed.
-
ChatGPT-UserOpenAI, fetches a page a person asked for
Allowed
Group
*every botAllow: /Allow: / (line 2) in the * group matches / and no Disallow rule does, so ChatGPT-User is allowed.
-
Claude-SearchBotAnthropic, AI search answers
Allowed
Group
*every botAllow: /Allow: / (line 2) in the * group matches / and no Disallow rule does, so Claude-SearchBot is allowed.
-
Claude-UserAnthropic, fetches a page a person asked for
Allowed
Group
*every botAllow: /Allow: / (line 2) in the * group matches / and no Disallow rule does, so Claude-User is allowed.
-
Perplexity-UserPerplexity, fetches a page a person asked for
Allowed
Group
*every botAllow: /Allow: / (line 2) in the * group matches / and no Disallow rule does, so Perplexity-User is allowed.
-
ApplebotApple, search results
Allowed
Group
*every botAllow: /Allow: / (line 2) in the * group matches / and no Disallow rule does, so Applebot is allowed.
-
Applebot-ExtendedApple, AI training
Allowed
Group
*every botAllow: /Allow: / (line 2) in the * group matches / and no Disallow rule does, so Applebot-Extended is allowed.
-
CCBotCommon Crawl, AI training
Allowed
Group
*every botAllow: /Allow: / (line 2) in the * group matches / and no Disallow rule does, so CCBot is allowed.
-
meta-externalagentMeta, AI training
Allowed
Group
*every botAllow: /Allow: / (line 2) in the * group matches / and no Disallow rule does, so meta-externalagent is allowed.
-
meta-externalfetcherMeta, fetches a page a person asked for
Allowed
Group
*every botAllow: /Allow: / (line 2) in the * group matches / and no Disallow rule does, so meta-externalfetcher is allowed.
-
AmazonbotAmazon, AI training
Allowed
Group
*every botAllow: /Allow: / (line 2) in the * group matches / and no Disallow rule does, so Amazonbot is allowed.
-
DuckDuckBotDuckDuckGo, search results
Allowed
Group
*every botAllow: /Allow: / (line 2) in the * group matches / and no Disallow rule does, so DuckDuckBot is allowed.
-
DuckAssistBotDuckDuckGo, fetches a page a person asked for
Allowed
Group
*every botAllow: /Allow: / (line 2) in the * group matches / and no Disallow rule does, so DuckAssistBot is allowed.
-
MistralAI-UserMistral, fetches a page a person asked for
Allowed
Group
*every botAllow: /Allow: / (line 2) in the * group matches / and no Disallow rule does, so MistralAI-User is allowed.
Problems in the file
Worth fixing1
These work, but are not right.
"EntityMap" is not a directive I recognise, so some crawlers will skip it.
EntityMap: https://sunnypatel.co.uk/entitymap.json
Also in the file
Sitemaps it lists
What to do
Line 9: "EntityMap" is not a directive I recognise, so some crawlers will skip it.
EntityMap: https://sunnypatel.co.uk/entitymap.json
How this check works
- For a live file, I read
/robots.txton your site once, as SitemapBot, the bot that runs this check. For pasted text, I fetch nothing. - I read every line, group the rules under their
User-agentlines, and list lines that are not rules or that I do not recognise. - For the path you give, each bot uses the group that names it most closely, or the
*group when none names it. - Within that group, the longest rule that matches the path decides. When an Allow and a Disallow rule are the same length, Allow wins.
- I show that line against every bot, and mark it in the file.
What it cannot know
- Whether a bot obeys the file. Each crawler makes its own call.
- Whether something in front of your site, such as a firewall, turns a bot away. The bot check asks your site as each bot and shows that.
- Whether pasted text matches the file your server sends. Test the live address to be sure.
- More than the first 100 KB of a very long file is not kept with the result, though every line is tested.
Questions
Why is this address blocked?
Type the address with its path, or change the path in the result. Each bot shows the one line that decides, the group it comes from and the reason in one sentence. Press the line to see it in your file.
Which group applies to Googlebot?
The group whose User-agent line names it most closely. If no group names it, the * group applies. The table shows the group each bot used.
What are the Cloudflare lines I did not write?
If your file has the BEGIN and END Cloudflare Managed content markers, I show which lines sit between them, so you can see which part of the file is inside that block.
What if my robots.txt answers 404 or a server error?
With a 404 or another 4xx answer other than 429, I treat it as no file, and every bot has no rule. With a server error or no answer, I cannot say what the file allows, so I test no bots.
Is my pasted text kept?
It goes only to my checker, which fetches nothing for it. Up to 100 KB of it is kept with the result for 90 days, so the result's link works.
How many tests can I run?
Ten tests of a live file an hour from one connection, and 60 tests of pasted text.