Skip to main content

Web Tools

Robots.txt Tester

Check robots.txt rules for a crawler and path.

Robots.txt Tester tool

Loading tool…

Jump to a section

About This Tool

Robots.txt controls crawler access before a search engine or bot requests pages from a site. A small rule such as Disallow: /admin/ or Allow: /private/public/ can affect crawling, indexing workflows, and deployment checks. This tester lets you paste robots.txt content, choose a crawler user agent, and test a URL path locally so you can understand which rule applies before changing a production file.

How To Use It

  1. Paste the robots.txt rules you want to test.
  2. Enter a crawler user agent such as Googlebot, Bingbot, or *.
  3. Enter a URL path beginning with / or a full URL; the tool tests the path and query portion.
  4. Review whether the path is allowed, which rule matched, and which rules applied to that crawler.

Examples

Block private sections

Use Disallow: /private/ to block crawler access to paths beginning with /private/. The tester reports that /private/report is blocked.

Allow a narrower public path

If /private/ is disallowed but /private/public/ is allowed, the longer matching Allow rule permits /private/public/page.

Check file patterns

A rule such as Disallow: /tmp/*.pdf$ blocks PDF files under /tmp/ when the path ends with .pdf, while other file names remain allowed.

Useful Notes

Longest matching rule wins

When multiple Allow and Disallow rules match, the most specific matching path pattern by canonical path length in octets wins. Raw UTF-8 and equivalent percent-encoded paths are compared consistently. If an Allow and Disallow rule have the same matching length, this tester treats Allow as the winner.

User-agent group precedence

When a crawler matches both a specific group such as Googlebot and the fallback * group, this tester evaluates the most specific matching user-agent group. Separate groups with the same matching user agent are combined.

No match means allowed

Robots.txt is permissive by default. If no applicable rule blocks a crawler for a path, the path is considered allowed by the robots rules.

Robots.txt is not indexing control

Blocking crawling is different from removing a URL from search results. Pages can still appear in limited ways if other pages link to them. Use appropriate noindex behavior where indexing control is needed.

Local parser scope

The tool handles common User-agent, Allow, Disallow, Sitemap, * wildcard, and $ end-anchor behavior. It does not fetch a live file, submit anything to search engines, or guarantee every crawler-specific extension.

FAQ

Does this tool fetch my live robots.txt file?

No. Paste the content you want to test. The tool runs locally and does not contact your website or a search engine.

Should I use robots.txt to hide private pages?

No. Robots.txt is public and cooperative. Sensitive pages should require authentication or proper access controls, not just crawler blocking.

What happens if both Allow and Disallow match?

The longest matching path rule wins. If both matches are the same length, Allow is treated as more specific for this tester.

Does Disallow prevent indexing?

Not always. It prevents compliant crawlers from fetching blocked URLs, but indexing behavior can depend on links, cached knowledge, and search engine policies.