Robots.txt Tester
Test a URL against pasted robots.txt rules. See the matching line and understand what it means.
A rule check you can follow.
Paste the actual plain-text rules you want to inspect, enter an absolute HTTP or HTTPS URL, and select the crawler token. The checker shows whether those rules allow the URL and which line matched. An empty file or a URL without a matching restriction is allowed by the supplied rules.
Read the result in context
A specific user-agent group can override the generic group. Matching paths are case-sensitive. The supported matching includes wildcard * and end-of-path $. A more specific matching rule takes precedence; Allow wins an equivalent Allow/Disallow tie. Groups for the same agent are combined by the parser.
What this cannot prove
This tool does not fetch your website, verify bot identity, inspect Cloudflare or another firewall, read noindex/canonical metadata, or confirm indexation. It checks the pasted rules against the URL. Sitemap, crawl-delay and other directives do not determine this allow/disallow result.
Sources & method
Robots Exclusion Protocol · RFC 9309 ↗
The tool performs a local check on the information supplied. Review the provider documentation before changing a live website.