1
User-agent rules
Generated robots.txt
0
Build a robots.txt file with custom rules per crawler, allow and disallow paths, crawl-delay, and sitemap links. Preview the result live and download it ready to upload to your site's root.
Start building robots.txt1
0
robots.txt is a plain text file that tells search engine crawlers which URLs they may or may not request from your site. This tool lets you build one or more user-agent rule groups with allow and disallow paths, an optional crawl-delay, and sitemap links, then preview the exact file live before downloading it.
Add separate rule sets for specific bots alongside a default group for everyone else.
List the paths each crawler should skip or is explicitly allowed to access.
Point crawlers straight to your sitemap so new pages get discovered faster.
Start from Allow all, Block all, or WordPress-friendly defaults in one click.
The file is generated locally in your browser. Nothing is uploaded.
Copy the text or download it as a ready-to-upload robots.txt file.
| Goal | Rule |
|---|---|
| Block an admin area from all bots | Disallow: /admin/ under User-agent: * |
| Allow one file inside a blocked folder | Allow: /private/public-file.pdf |
| Point crawlers to your sitemap | Sitemap: https://example.com/sitemap.xml |
A robots.txt file tells search engine crawlers which parts of a website they are allowed or not allowed to request. It lives at the root of a domain, such as example.com/robots.txt.
No. Disallowing a URL in robots.txt only blocks crawling, not indexing. A blocked page can still appear in search results if it is linked from elsewhere. Use a noindex meta tag to prevent indexing.
The asterisk is a wildcard that matches every crawler. Rules under User-agent: * apply to all bots unless a more specific user-agent group overrides them.
Yes. Adding a Sitemap directive gives crawlers a direct path to your full list of URLs, which can help new or updated pages get discovered faster.
Googlebot ignores the Crawl-delay directive, though it is still respected by some other search engines like Bing and Yandex. Use Google Search Console to adjust Google's crawl rate instead.
Upload it to the root directory of your domain so it is reachable at https://yourdomain.com/robots.txt. It will not work if placed in a subfolder.
No. The file is assembled entirely in your browser using client-side JavaScript. Nothing you enter is sent to a server.