sitemap.bot

What does a bot get from your page?

Ask for one page as GPTBot, Googlebot or another crawler. See every redirect, the answer, and how it compares with an ordinary browser.

The sitemap bot knocking at your site's door while the check runs

One page, one bot, a plain answer. Free, no sign-up. Made by Sunny Patel.

Example A check I ran on 19 November 2026 for https://thebestmowers.co.uk/

Your site answered GPTBot with 403, but an ordinary browser was let in. The answer came through Cloudflare.

The sitemap bot with his arms crossed: bots were turned away

I asked for this one page using the bot's name and read the HTML. I do not run JavaScript, and the real bot can get a different answer.

Checked on 19 November 2026 at 18:53 GMT.

Asked as GPTBot (OpenAI), with this user agent: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot

The way in

RefusedHTTP 403Final answer to GPTBot, in 46 ms

  1. Answerhttps://thebestmowers.co.uk/RefusedHTTP 403

What GPTBot got, beside a browser

Different. As GPTBot I got 403. As an ordinary browser I got 200, so the site treats them differently.

What GPTBot got, beside what an ordinary browser got
FactGPTBotAn ordinary browser
StatusRefusedHTTP 403Let inHTTP 200
TitleNot readThe Best Mowers UK: Lawn Mower Reviews & Buying Guides
Words readNot read484

What to do

  1. In Cloudflare, open the AI bot blocking setting (Security, then Bots) and check whether AI bots are blocked.

  2. Check your firewall rules (Security, then WAF) for any rule that blocks or challenges by user agent or by bot score.

  3. Run the check again after changing a setting. Each bot you want to let in needs to be allowed, not only the first one.

Evidence in the answer

The headers I look for, as your site sent them to GPTBot. They name Cloudflare in front of your site.

server
cloudflare
cf-mitigated
Not sent
cf-ray
Present
x-sucuri-id
Not sent
x-akamai
Not sent
via
Not sent
Challenge page
No

robots.txt for /

Allowedby Allow: / (line 10, in the gptbot group)

I read robots.txt (opens in a new tab) first (HTTP 200). See every bot for this path in the robots.txt tester

How this check works

  1. I read your robots.txt once, as SitemapBot, the bot that runs this check. If it tells SitemapBot or the bot you picked to stay out of the page, I stop and show the rule.
  2. I ask for the page using the bot's name, and follow each redirect myself, so you see the status at every step.
  3. If the first answer asks me to wait, I ask once more.
  4. Unless you picked an ordinary browser, I ask once more as an ordinary browser, and compare the status, the title and how many words each got.
  5. I read the HTML the server sends, up to 256 KB, and count the words and links in it.

What it cannot know

  • What the real bot gets. It comes from its own company's addresses, and a firewall can treat it differently from a request that only uses its name.
  • Anything a page builds with JavaScript. I do not run it.
  • The browser's canonical, meta robots and links. For the browser I keep only the status, the title and the word count.
  • Any page but this one. The bot check asks for your home page as every crawler at once.

Questions

Does Googlebot see the same page as me?

Pick Googlebot and check the page. I compare what came back for Googlebot's name with what an ordinary browser got: the status, the title and the words. The real Googlebot can still get a different answer.

Why does the bot get less text?

Either the page builds its text with JavaScript, which this check does not run, or something in front of the site sends bots a lighter page. Open the page with JavaScript off, or view its source, to see which.

Why was nothing fetched?

Your robots.txt tells that bot, or SitemapBot, to stay out of the page. I follow it, so I did not ask. The result shows the rule and its line.

Does it run JavaScript?

No. I read the HTML your server sends, up to 256 KB.

Is this the same as URL Inspection in Search Console?

No. Search Console shows what Google itself fetched. This shows what your site sends, from my server, to a request that uses a bot's name.

How many checks can I run?

Ten an hour from one connection. Each result has its own link, which works for 90 days.