Intelmark

How to Create a robots.txt File (Beginner's Guide)

A robots.txt file is a plain text file at the root of your website that gives crawlers instructions about which URLs they may visit. Search engines check it before they crawl your site.

The basic syntax

User-agent: *
Disallow: /admin/
Allow: /admin/public/

Sitemap: https://www.example.com/sitemap.xml
  • User-agent names the crawler the rules apply to. An asterisk means all crawlers.
  • Disallow blocks a path. An empty Disallow allows everything.
  • Allow makes an exception inside a blocked folder.
  • Sitemap gives the full URL of your XML sitemap.

Where to put it

Upload the file to the root of your domain so it loads at https://yourdomain.com/robots.txt. Each subdomain needs its own file.

Create one in seconds

Use our Robots.txt Generator: choose your rules, add your sitemap and copy the result.

What robots.txt cannot do

  • It is not security. It is a request. Malicious bots ignore it, and the file itself is public.
  • It does not remove pages from Google. A blocked URL can still appear in results if other pages link to it. To keep a page out of search, use a noindex meta tag (build one with the Meta Tag Generator) and make sure the page is not blocked, so crawlers can see the tag.
  • Do not block CSS and JavaScript that your pages need to render. Google needs them to understand your layout.

Common mistakes

  1. Leaving Disallow: / in place after launching a site from staging.
  2. Using wrong case in paths. Paths are case-sensitive.
  3. Forgetting the sitemap line.

After uploading, test the file in Google Search Console.