What does Disallow: / mean?

It tells crawlers not to fetch any page on the site. Use it only on staging or private sites, never on a live site you want found in search.

Is robots.txt a way to hide private pages?

No. The file is public and only polite crawlers follow it. Protect private pages with a login, not with robots.txt.

Do AI crawlers have to obey robots.txt?

The well-known ones say they do, but it is a request, not a lock. The list of names changes, so check each company's own page for the current name.

robots.txt generator: allow, block and add your sitemap

Create a robots.txt that tells search engines what to crawl, with a sitemap line and optional AI crawler blocks.

For any website. The file must sit at the top of the domain, not in a subfolder.

robots.txt is a small text file at the top of your site that tells search engine crawlers which paths to skip. Choose your rules below and upload the result as robots.txt so it opens at yourdomain.com/robots.txt.

Create your config file

Type your own values. The file below updates as you type, and nothing you type leaves your browser.

/robots.txt

User-agent: *
Allow: /
Disallow: /admin/

Sitemap: https://example.com/sitemap.xml

How this setup works

Diagram: Crawler, then robots.txt, then Allowed pages — Crawler (visits your site) → robots.txt (read first) → Allowed pages (are crawled)

Step-by-step setup

Save the lines above in a file named robots.txt (all lowercase).
Upload it to the top folder of your site, so it opens at yourdomain.com/robots.txt.
Open that address in your browser to check it shows the text.
In Google Search Console, open the robots.txt report to see that Google read it without errors.
Re-check it after a site change, since one wrong Disallow line can hide the whole site.

What the important lines mean

User-agent: * means the rules below apply to every crawler.
Disallow: / blocks the whole site. Disallow: /admin/ blocks only that folder.
Allow makes an exception inside a blocked folder.
Sitemap: gives crawlers the address of your sitemap. It needs the full https:// address.

Common errors and how to fix them

My page is still in Google after I blocked it: robots.txt stops crawling, not listing. A blocked page can still appear if other sites link to it. To remove a page, let it be crawled and add a noindex tag, or remove it in Search Console.
Blocked by robots.txt in Search Console: A Disallow line matches that page. Check for Disallow: /, which was meant for a staging site, left on a live one.
robots.txt returns 404: The file is not in the top folder, or the name has capital letters. It must be exactly robots.txt at the root of the domain.
Sitemap could not be read: The Sitemap line needs the full address starting with https:// and the sitemap must open in a browser.

Frequently asked questions

What does Disallow: / mean?: It tells crawlers not to fetch any page on the site. Use it only on staging or private sites, never on a live site you want found in search.
Is robots.txt a way to hide private pages?: No. The file is public and only polite crawlers follow it. Protect private pages with a login, not with robots.txt.
Do AI crawlers have to obey robots.txt?: The well-known ones say they do, but it is a request, not a lock. The list of names changes, so check each company's own page for the current name.

Written for: Google and Bing robots.txt rules · Updated 2026-10-11

Free tools that help

Text diff checkerCompare two texts or files line by line and see what was added, removed or changed, with changed words highlighted. Side-by-side or one column.

More server configs

nginx redirect www to non-wwwSend www.example.com to example.com (or the reverse) with one permanent redirect.
nginx load balancer configSplit traffic across two or three copies of your app with an nginx upstream block.
nginx rate limit configSlow down abusive visitors with limit_req: a request rate per IP, a burst, and a 429 answer.
php.ini settingsSet the PHP values most sites need: upload size, memory, run time, errors, time zone and OPcache.
MySQL my.cnf settingsA small MySQL or MariaDB override file for connections, memory, character set and the slow query log.
nginx reverse proxy configSend visitors from port 80 to an app running on port 3000.

See all 41 config generators