robots.txt Builder
Visually build, parse, and test robots.txt files with rule matching that follows Google's spec.
Quick templates
Start from a common pattern, then customize.
User-agent groups
Each group declares rules for one or more crawlers.
Group
User-agents
*
Allow0
No rules yet.
Disallow0
No rules yet.
Crawl-delay (seconds)
Ignored by Google. Bing and Yandex honor it.
Sitemaps
Absolute sitemap URLs declared at the file level.
No sitemaps yet.
Generated robots.txt
This is what crawlers will see at /robots.txt.
1 lines14 B / 500 KB
User-agent: *
Validation
Issues are live-updated as you edit.
Group for "*" has no rules.