Free · No Account Required
Free Robots.txt Tester
Paste a URL, choose a crawler, and see whether robots.txt allows or blocks that path.
What the robots.txt tester checks
Use this free tool to test a public URL against the site's robots.txt file for Googlebot, Bingbot, GPTBot, or a wildcard crawler.
Crawler-specific verdict
See whether the selected user agent is allowed or blocked for the exact path you entered.
Matching rule
Review the Allow or Disallow directive, user-agent group, and line number that produced the verdict.
Robots context
Check sitemap declarations, crawl-delay hints, and a readable excerpt of the robots.txt file.
Why robots.txt matters for SEO
Important pages can be blocked accidentally
A single Disallow rule can keep search engines away from product pages, blog posts, or entire folders.
Different crawlers can see different rules
Googlebot, Bingbot, AI crawlers, and generic bots may match separate user-agent groups.
Sitemaps and crawl hints live here too
Robots.txt often points crawlers to XML sitemaps and may include crawl-delay directives.
How to use the result
Allowed
The selected crawler is not blocked by robots.txt for this URL.
Blocked
Review the matching rule and update robots.txt if the URL should be crawlable.
Unavailable
Fix the robots.txt response first. Crawlers may delay or avoid crawling when robots.txt cannot be read.
Audit crawl access across every important page
The free tester checks one URL at a time. SEO Core crawls your whole site, flags blocked pages, and connects robots issues to the pages that need attention.