Robots.txt Generator
Build a clean robots.txt file to tell search engine crawlers which parts of your site they can visit. Add your sitemap and, if you wish, block AI training bots.
Robots.txt Generator
How to use the Robots.txt Generator
- Choose whether to allow or block crawlers by default.
- List any folders or URLs to block, and add your sitemap URL.
- Press "Generate robots.txt", save it as robots.txt and upload it to your site root.
About this tool
The robots.txt file sits at the root of your domain and gives crawlers instructions about which paths they may crawl. It is a request, not a security measure: well-behaved bots follow it, but it does not protect private data.
Do not use robots.txt to remove a page from Google. Blocked pages can still be indexed if other sites link to them. To keep a page out of results, use a noindex meta tag or password protection instead. Add your sitemap with the Sitemap directive so crawlers can find your pages faster. The AI-crawler option adds rules for well-known training bots, which is a polite request rather than a technical block.
Frequently asked questions
Where do I put the robots.txt file?
In the root folder of your domain, so it loads at https://yourdomain.com/robots.txt.
Does robots.txt stop pages from showing in Google?
Not reliably. Use a noindex tag to keep a page out of results.
Does Google use the crawl delay?
No. Googlebot ignores Crawl-delay. Some other crawlers, such as Bing, respect it.
Should I block AI crawlers?
That is your call. Blocking asks AI companies not to use your content for training, but it is voluntary on their side.