SEO Free · no signup

Robots.txt Tester

Fetch and analyze robots.txt rules. Test which paths are allowed or blocked for crawlers.

The Robots.txt Tester downloads a site's /robots.txt file and shows exactly how crawlers will interpret it: which user-agents have rules, which paths are Disallowed or Allowed, where the sitemaps are declared, and any crawl-delay. Paste a URL path and it tells you whether that path is blocked or crawlable under the current rules. Robots.txt controls crawling, not indexing — a Disallowed page can still appear in search results without a snippet if other sites link to it — so use this tool to confirm you are blocking crawl budget waste (faceted URLs, admin paths, search results) while never accidentally blocking pages, CSS, or JS you need indexed and rendered.

Updated Krawly Editorial TeamIn-house engineers, writers & reviewers

Explore More Free Tools

Discover 150+ free tools for web scraping, SEO analysis, OSINT, and more. 30 free uses every day — no signup required.

Explore All Tools
150+ Free Tools No Signup Required JSON / CSV / Excel 30 Uses / Day
Quick answer

The Robots.txt Tester downloads a site's /robots.txt file and shows exactly how crawlers will interpret it: which user-agents have rules, which paths are Disallowed or Allowed, where the sitemaps are declared, and any crawl-delay. Paste a URL path and it tells you whether that path is blocked or crawlable under the current rules. Robots.txt controls crawling, not indexing — a Disallowed page can still appear in search results without a snippet if other sites link to it — so use this tool to confirm you are blocking crawl budget waste (faceted URLs, admin paths, search results) while never accidentally blocking pages, CSS, or JS you need indexed and rendered.

What is Robots.txt Tester?

The Robots.txt Tester fetches the robots.txt file from any domain's root and parses it into readable rules — grouping the Allow and Disallow directives by user-agent, listing every declared Sitemap URL, and surfacing any crawl-delay settings. You can then test specific paths against those rules to see instantly whether Googlebot (or any crawler) would be allowed to fetch them. Because a single misplaced Disallow line can hide an entire section of your site from search engines, this tool turns an easy-to-misread text file into a clear, debuggable map of your crawl directives.

How to use Robots.txt Tester

  1. 1

    Enter the domain

    Paste any URL from the site — the tool automatically requests the robots.txt file from the domain root (https://example.com/robots.txt), since robots.txt only ever lives at the root.

  2. 2

    Review the parsed rules

    Read the directives grouped by user-agent. The User-agent: * block applies to all crawlers unless a more specific block (like User-agent: Googlebot) overrides it.

  3. 3

    Test a specific path

    Type a path such as /blog/my-post or /checkout and the tool reports whether it is Allowed or Disallowed under the current rules, so you can confirm important pages are crawlable.

  4. 4

    Grab the sitemap and fix issues

    Copy any declared Sitemap URL for your audit, and if you spot an over-broad Disallow, edit robots.txt on your server, redeploy, and re-run the test.

Try it when you need to…

  • Try it when pages vanished from Google after a deploy and you suspect a new Disallow rule is the cause
  • Try it when you need to find a site's sitemap but there is no obvious link to it on the page
  • Try it when you want to confirm that a sensitive path like /cart or /wp-admin is actually blocked from crawlers

Use cases

  • Confirm that a Disallow rule meant to block /admin or /cart isn't accidentally catching a public path because of a missing trailing slash or an overly broad wildcard
  • Find the sitemap URL for an unfamiliar site by reading the Sitemap: lines the robots.txt tester extracts, then feed those into a crawler or audit
  • Diagnose why a whole section of a site stopped appearing in Google after a launch by testing its paths against the live robots rules
  • Verify that you are not blocking CSS and JavaScript directories, which prevents Google from rendering the page and can hurt mobile rankings
  • Audit a competitor's robots.txt to learn which URL patterns they deliberately keep out of the index

Key features

Full parsing of Allow, Disallow, and Crawl-delay directives grouped per user-agent block
Automatic extraction of every Sitemap: declaration in the file
Path testing that reports allowed or blocked for any URL you enter, matching how crawlers apply the most-specific rule
Raw file view with one-click copy so you can inspect the exact syntax
Detection of common mistakes such as blocking the entire site with Disallow: /

Tips & best practices

robots.txt blocks crawling, not indexing. A Disallowed URL that other sites link to can still show up in Google as a bare link with no description. To truly keep a page out of the index, allow crawling and use a noindex meta robots tag instead.

Never Disallow your CSS or JS folders. Google renders pages to judge mobile-friendliness and layout, and blocking those resources makes it see a broken page, which can lower rankings.

Rule matching is prefix-based and the most specific rule wins. Disallow: /search matches /search, /search-results, and /searching — add a trailing slash or use Allow to carve out exceptions when the prefix is too greedy.

robots.txt must be UTF-8 and served at the exact root over the same protocol and host; a robots.txt at a subdirectory or on a redirecting host is ignored by crawlers, so always test the canonical https root.

Frequently asked questions

Robots.txt is a plain text file at your domain root that tells crawlers which URLs they may or may not fetch. It uses User-agent lines to target specific bots and Allow/Disallow lines to permit or block path prefixes. Compliant crawlers like Googlebot read it before crawling, but it is a voluntary standard — malicious bots can ignore it entirely.

No. Robots.txt only prevents crawling. If a Disallowed page is linked from elsewhere, Google can still index the URL itself (shown as a link with no snippet, because it wasn't allowed to read the content). To reliably keep a page out of search results, let crawlers fetch it and add a <meta name="robots" content="noindex"> tag or an X-Robots-Tag header.

A single Disallow: / under User-agent: * blocks the entire site from all crawlers — nothing gets crawled. This is common on staging sites but catastrophic if it leaks to production, because it can wipe your rankings within days. The tester flags this pattern so you catch it before it does damage.

Google renders pages like a browser to evaluate layout, mobile-friendliness, and content that loads via scripts. If your robots.txt Disallows the folders holding CSS or JS, Googlebot sees an unstyled, broken version of the page, which can lower its quality assessment and hurt rankings. Keep resource directories crawlable.

Yes. Create separate User-agent blocks — for example User-agent: Googlebot with its own rules, then User-agent: Bingbot, then a catch-all User-agent: * for everyone else. Each crawler obeys the most specific block that names it and ignores the others. The tester groups the rules by user-agent so you can verify each bot's directives.

It must be at the root of the host, exactly at /robots.txt, served over the protocol and subdomain the crawler is visiting. https://example.com/robots.txt governs that host; a file at https://example.com/folder/robots.txt is ignored, and https://blog.example.com needs its own robots.txt. That is why the tool always requests it from the domain root.

Crawl-delay asks crawlers to wait a set number of seconds between requests to reduce server load. Bing and some other crawlers respect it, but Googlebot ignores crawl-delay entirely — you control Google's crawl rate through Search Console settings instead. The tester surfaces any crawl-delay value so you know which bots it will actually affect.

You might also like

Related tools & similar free utilities

View all 27 SEO tools
SEO

Link Checker

Crawl any webpage and check all links for broken URLs, redirects, timeouts, and errors.

Use Tool
SEO

OG Preview

Preview how your page looks when shared on social media — extract OG tags, Twitter Cards, and get an SEO score.

Use Tool
SEO

Redirect Trace

Trace the full HTTP redirect chain for any URL — see 301/302/307/308 hops, status codes, and final destination.

Use Tool
SEO

Link Mapper

Map all internal and external links on a page — anchor text, nofollow tags, link depth distribution, and orphan detection.

Use Tool
Analysis

Robots.txt

Fetch and analyze robots.txt rules, sitemaps, and crawler directives.

Use Tool
Developer

Regex Test

Test regular expressions against text with match highlighting, groups, and flags support.

Use Tool

Looking for something specific? Browse all 158 Krawly tools