Web Tools
Robots.txt Tester
Check robots.txt rules for a crawler and path.
Allowed
Allowed by the longest matching Allow rule.
Tested /private/public/page for Googlebot.
Matched rule
Line 4: allow: /private/public/Applicable rules for this crawler (4)
| Line | Agents | Directive | Path |
|---|---|---|---|
| 2 | * | disallow | /admin/ |
| 3 | * | disallow | /private/ |
| 4 | * | allow | /private/public/ |
| 5 | * | disallow | /tmp/*.pdf$ |
Parsed summary
4 rules and 1 sitemap entry found.
Sitemaps: https://example.com/sitemap.xml
This tester runs locally and follows common robots.txt matching rules. It does not fetch your live robots.txt, verify crawler behavior, or replace checks in Search Console or server logs.
About This Tool
Robots.txt controls crawler access before a search engine or bot requests pages from a site. A small rule such as Disallow: /admin/ or Allow: /private/public/ can affect crawling, indexing workflows, and deployment checks. This tester lets you paste robots.txt content, choose a crawler user agent, and test a URL path locally so you can understand which rule applies before changing a production file.
How To Use It
- Paste the robots.txt rules you want to test.
- Enter a crawler user agent such as Googlebot, Bingbot, or *.
- Enter a URL path beginning with / or a full URL; the tool tests the path and query portion.
- Review whether the path is allowed, which rule matched, and which rules applied to that crawler.
Examples
Block private sections
Use Disallow: /private/ to block crawler access to paths beginning with /private/. The tester reports that /private/report is blocked.
Allow a narrower public path
If /private/ is disallowed but /private/public/ is allowed, the longer matching Allow rule permits /private/public/page.
Check file patterns
A rule such as Disallow: /tmp/*.pdf$ blocks PDF files under /tmp/ when the path ends with .pdf, while other file names remain allowed.
Useful Notes
Longest matching rule wins
When multiple Allow and Disallow rules match, the most specific matching path pattern wins. If an Allow and Disallow rule have the same matching length, this tester treats Allow as the winner.
No match means allowed
Robots.txt is permissive by default. If no applicable rule blocks a crawler for a path, the path is considered allowed by the robots rules.
Robots.txt is not indexing control
Blocking crawling is different from removing a URL from search results. Pages can still appear in limited ways if other pages link to them. Use appropriate noindex behavior where indexing control is needed.
Local parser scope
The tool handles common User-agent, Allow, Disallow, Sitemap, * wildcard, and $ end-anchor behavior. It does not fetch a live file, submit anything to search engines, or guarantee every crawler-specific extension.
FAQ
Does this tool fetch my live robots.txt file?
No. Paste the content you want to test. The tool runs locally and does not contact your website or a search engine.
Should I use robots.txt to hide private pages?
No. Robots.txt is public and cooperative. Sensitive pages should require authentication or proper access controls, not just crawler blocking.
What happens if both Allow and Disallow match?
The longest matching path rule wins. If both matches are the same length, Allow is treated as more specific for this tester.
Does Disallow prevent indexing?
Not always. It prevents compliant crawlers from fetching blocked URLs, but indexing behavior can depend on links, cached knowledge, and search engine policies.
Related Tools
HTTP Status Code Lookup
Find HTTP status meanings and next steps.
URL Parser & Inspector
Inspect URL components and query parameters.
Cache-Control Header Builder
Build and understand HTTP Cache-Control directives.
HTTP Header Parser
Parse and inspect raw HTTP header fields locally.