What does a bot get from your page?
Ask for one page as GPTBot, Googlebot or another crawler. See every redirect, the answer, and how it compares with an ordinary browser.
One page, one bot, a plain answer. Free, no sign-up. Made by Sunny Patel.
Example A check I ran on 19 November 2026 for https://thebestmowers.co.uk/
Your site answered GPTBot with 403, but an ordinary browser was let in. The answer came through Cloudflare.

I asked for this one page using the bot's name and read the HTML. I do not run JavaScript, and the real bot can get a different answer.
Checked on 19 November 2026 at 18:53 GMT.
Asked as GPTBot (OpenAI), with this user agent: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot
The way in
RefusedHTTP 403Final answer to GPTBot, in 46 ms
- Answer
https://thebestmowers.co.uk/RefusedHTTP 403
What GPTBot got, beside a browser
Different. As GPTBot I got 403. As an ordinary browser I got 200, so the site treats them differently.
| Fact | GPTBot | An ordinary browser |
|---|---|---|
| Status | RefusedHTTP 403 | Let inHTTP 200 |
| Title | Not read | The Best Mowers UK: Lawn Mower Reviews & Buying Guides |
| Words read | Not read | 484 |
What to do
In Cloudflare, open the AI bot blocking setting (Security, then Bots) and check whether AI bots are blocked.
Check your firewall rules (Security, then WAF) for any rule that blocks or challenges by user agent or by bot score.
Run the check again after changing a setting. Each bot you want to let in needs to be allowed, not only the first one.
Evidence in the answer
The headers I look for, as your site sent them to GPTBot. They name Cloudflare in front of your site.
- server
cloudflare- cf-mitigated
- Not sent
- cf-ray
- Present
- x-sucuri-id
- Not sent
- x-akamai
- Not sent
- via
- Not sent
- Challenge page
- No
robots.txt for /
Allowedby Allow: / (line 10, in the gptbot group)
I read robots.txt (opens in a new tab) first (HTTP 200). See every bot for this path in the robots.txt tester
How this check works
- I read your robots.txt once, as SitemapBot, the bot that runs this check. If it tells SitemapBot or the bot you picked to stay out of the page, I stop and show the rule.
- I ask for the page using the bot's name, and follow each redirect myself, so you see the status at every step.
- If the first answer asks me to wait, I ask once more.
- Unless you picked an ordinary browser, I ask once more as an ordinary browser, and compare the status, the title and how many words each got.
- I read the HTML the server sends, up to 256 KB, and count the words and links in it.
What it cannot know
- What the real bot gets. It comes from its own company's addresses, and a firewall can treat it differently from a request that only uses its name.
- Anything a page builds with JavaScript. I do not run it.
- The browser's canonical, meta robots and links. For the browser I keep only the status, the title and the word count.
- Any page but this one. The bot check asks for your home page as every crawler at once.
Questions
Does Googlebot see the same page as me?
Pick Googlebot and check the page. I compare what came back for Googlebot's name with what an ordinary browser got: the status, the title and the words. The real Googlebot can still get a different answer.
Why does the bot get less text?
Either the page builds its text with JavaScript, which this check does not run, or something in front of the site sends bots a lighter page. Open the page with JavaScript off, or view its source, to see which.
Why was nothing fetched?
Your robots.txt tells that bot, or SitemapBot, to stay out of the page. I follow it, so I did not ask. The result shows the rule and its line.
Does it run JavaScript?
No. I read the HTML your server sends, up to 256 KB.
Is this the same as URL Inspection in Search Console?
No. Search Console shows what Google itself fetched. This shows what your site sends, from my server, to a request that uses a bot's name.
How many checks can I run?
Ten an hour from one connection. Each result has its own link, which works for 90 days.