Is this URL blocked for that crawler?
Test any path on a site against any crawler and see the exact robots.txt rule that decides it. Or paste a draft robots.txt and test it before you publish.
robots.txt rules, briefly
How does the tester decide?
As RFC 9309 describes: a crawler follows the group that names it (names are not case-sensitive), otherwise the group for *. Within the group, the longest matching rule wins, and Allow wins a tie. * matches anything and $ marks the end of the address.
What is the crawler name?
The token the crawler looks for in robots.txt, for example Googlebot, Bingbot, GPTBot or ClaudeBot. It is usually part of the crawler's user agent.
Does "allowed" mean the page will be indexed?
No. robots.txt only controls crawling. A page can be blocked from crawling and still appear in search results if other sites link to it; use a noindex tag to keep it out.
Why test a draft?
A single misplaced line in robots.txt can hide a whole site from search engines. Testing a draft against your key pages first is the safest way to change it.
More free checks
No sign-up, nothing stored. See all free tools.