Robots.txt Validator
Check your robots.txt syntax and find crawling issues before search engines do. Paste your file or load a sample, then see valid directives, warnings, and errors line by line, no code, no signup.
Enter a URL to automatically fetch and validate its robots.txt file.
Paste your robots.txt content or load a sample to validate
Validation runs entirely in your browser — pasted content never leaves your device. Fetching a URL routes through our server to bypass CORS.
How it works
Validate your robots.txt in three steps
Paste or load your robots.txt
Drop in the contents of your site's robots.txt file, or click Load Sample to see a working example with crawler rules and sitemaps.
Read the results
We parse every line in your browser, group the rules by user-agent, and flag errors, warnings, and sitemap entries so you can see exactly how crawlers will read your file.
Fix and share
Patch the lines we flag and re-check until it's clean, then copy the file or share a link that reproduces your input for a teammate to review.
Common questions about robots.txt
A robots.txt file is a plain text file at the root of your site that tells search engine crawlers which paths they may or may not request. It uses simple directives, User-agent to target a crawler, then Allow and Disallow rules, and can point to your sitemaps. It's the first thing most crawlers read before they index your site.
The widely supported directives are User-agent (which crawler the following rules apply to), Disallow and Allow (paths to block or permit), Sitemap (the full URL of a sitemap), and Crawl-delay (a hint some crawlers honour, though Google ignores it). Lines starting with # are comments. This validator recognises each of these and warns on anything unknown or malformed.
Not reliably. Disallow stops compliant crawlers from fetching a page, but a blocked URL can still be indexed if other pages link to it, Google may show it without a description. To keep a page out of search results, use a noindex meta tag or header and leave the page crawlable, rather than blocking it in robots.txt.
It must live at the root of your domain, at /robots.txt, for example https://example.com/robots.txt. Crawlers only look there; a robots.txt in a subdirectory is ignored. Each subdomain needs its own file.
No. The validator parses your robots.txt entirely in your browser. Nothing you paste is sent to a server or stored, your data never leaves your device.
Still have questions? Contact us
More free tools
Crawlers see exactly what they should
Powerblog generates an optimized robots.txt with your sitemap included, so search engines crawl your content efficiently and you never have to hand-tune crawler directives again.
Try Powerblog free →Free for 7 days • No credit card • Cancel anytime