Skip to main content
CalcChef

Robots.txt Generator

Build a robots.txt from a preset such as Block AI crawlers or WordPress, or block and allow folders for named bots, with Sitemap lines and every path checked. How it works

Use the Robots.txt Generator

Replaces the groups below; sitemap lines are kept.

Group 1
Pick crawlers for group 1
Search engines
SEO tools
AI crawlers
Social

Each path starts with /. * matches any characters and $ marks the end of the URL. The longest matching rule wins.

Full addresses starting with https:// or http://.

Generated robots.txt

User-agent: *
Disallow: /admin

robots.txt controls crawling, not indexing, and anyone can read it. Upload the file to the top level of your site as /robots.txt. Google ignores Crawl-delay, so this tool does not write it.

About the Robots.txt Generator

The robots.txt generator starts with two lines, User-agent: * and Disallow: /admin, which ask every crawler to skip URLs whose path starts with /admin. It writes one block per group of user agents, one Allow or Disallow line per path, and Sitemap lines at the end. Start from a preset, Allow all, Block all, Block AI crawlers or WordPress, or tick crawlers by name from a list of 28. Copy the result into a file served at the top level of the site, such as https://example.com/robots.txt.

Example groups and the file they give

User agentsAllowDisallowrobots.txt
*(none)/adminUser-agent: * ⏎ Disallow: /admin
*(none)(none)User-agent: * ⏎ Disallow:
*/admin/public//admin/User-agent: * ⏎ Allow: /admin/public/ ⏎ Disallow: /admin/
Googlebot, Bingbot(none)/search ⏎ /*.pdf$User-agent: Googlebot ⏎ User-agent: Bingbot ⏎ Disallow: /search ⏎ Disallow: /*.pdf$
GPTBot(none)/User-agent: GPTBot ⏎ Disallow: /

⏎ marks a new line. An empty Disallow: allows everything, and the generator adds a note saying so.

Paths the generator refuses

PathMessage
admin/"admin/" must start with /. A path such as admin/ is ignored by Google; write /admin/.
*.pdf"*.pdf" must start with /, not *. RFC 9309 paths start with /; the * wildcard still works after it. Write /*.pdf.
/a#b"/a#b" contains #, which starts a comment in robots.txt. Write it as %23.
/a b"/a b" contains a space. Write it as %20.

Write one path per line. A comma is part of the path, so /a,/b is a single rule.

The presets and the file they write

Presetrobots.txt
Allow allUser-agent: * ⏎ Disallow:
Block allUser-agent: * ⏎ Disallow: /
Block AI crawlersUser-agent: GPTBot ⏎ User-agent: OAI-SearchBot ⏎ User-agent: ChatGPT-User ⏎ User-agent: ClaudeBot ⏎ User-agent: Claude-SearchBot ⏎ User-agent: Claude-User ⏎ User-agent: PerplexityBot ⏎ User-agent: Perplexity-User ⏎ User-agent: Google-Extended ⏎ User-agent: Applebot-Extended ⏎ User-agent: CCBot ⏎ User-agent: meta-externalagent ⏎ User-agent: Amazonbot ⏎ Disallow: /
WordPressUser-agent: * ⏎ Allow: /wp-admin/admin-ajax.php ⏎ Disallow: /wp-admin/

A preset replaces the groups and keeps the Sitemap lines. ⏎ marks a new line.

Frequently asked questions

How do I block all crawlers from my whole site?
Use the user agent * with the Disallow path /, which gives User-agent: * and Disallow: /. That asks every crawler that follows robots.txt to stay away. It does not remove pages that are already in search results, and a bot that ignores the file is not stopped by it.
What does an empty Disallow: mean?
It allows everything. A group with no Allow or Disallow paths is written as its User-agent lines and an empty Disallow:, and the generator adds a note saying so.
Does robots.txt keep a page out of Google?
No. It controls crawling. Google can index a blocked URL and show it without a snippet if other pages link to it, and anyone can read the file. Use a noindex tag instead, and leave the page crawlable so Google can see the tag.
Does Disallow: /admin also block /administrator?
Yes. A path matches every URL that starts with it, so Disallow: /admin blocks /admin, /admin.html, /administrator and everything under /admin/. Write /admin/ with the trailing slash to block only what is inside the folder; /admin itself then stays crawlable. Matching is case-sensitive, so /Admin is not blocked by either rule.
Which rule wins when Allow and Disallow both match?
The longest matching rule wins (RFC 9309). With Allow: /admin/public/ and Disallow: /admin/, /admin/public/report.html may be crawled and /admin/users.html may not.
How do I block AI crawlers such as GPTBot?
Pick Block AI crawlers under Start from a preset. It writes one group of 13 user agents with Disallow: /, each name checked on its operator’s page: GPTBot, OAI-SearchBot and ChatGPT-User (OpenAI), ClaudeBot, Claude-SearchBot and Claude-User (Anthropic), PerplexityBot and Perplexity-User, Google-Extended, Applebot-Extended, CCBot, meta-externalagent and Amazonbot. Google-Extended and Applebot-Extended do not crawl: Google and Apple say they decide whether pages their search crawlers fetch may be used for AI models, and blocking them does not affect search. ChatGPT-User, Claude-User and Perplexity-User fetch a page when a person asks, and OpenAI says robots.txt rules may not apply to those requests.

What robots.txt can and cannot do

Site owners use robots.txt to ask crawlers to skip areas such as /admin, internal search results or print versions, to give one crawler different rules from the rest, and to point crawlers at the sitemap. Start from a preset (Allow all, Block all, Block AI crawlers or WordPress), or tick crawlers by name from a list of 28 in four groups: search engines, SEO tools, AI crawlers and social. Copy the result or download it as robots.txt, ready to upload to the top level of the site. The file is public and only asks: it does not hide a page, and it does not keep a page out of search results.

Groups, paths and the longest match

A file is a list of groups. Each group is one or more User-agent: lines followed by its Allow: and Disallow: lines, one path per line. Every path starts with /; inside a path * matches any characters and $ marks the end of the URL. When an Allow and a Disallow rule both match a URL, the longest one wins (RFC 9309). A group with no paths gets an empty Disallow:, which allows everything. Sitemap lines come last and must be full http or https addresses.

Example: user agent *, Allow /admin/public/, Disallow /admin/ and the sitemap https://example.com/sitemap.xml give:

  • User-agent: *

  • Allow: /admin/public/

  • Disallow: /admin/

  • Sitemap: https://example.com/sitemap.xml

with a blank line before the Sitemap line. /admin/public/report.html may then be crawled and /admin/users.html may not. Google ignores Crawl-delay, so the generator does not write it.

Related Tools

View all tools
Web

Slug Generator

Turn a title into a lowercase URL slug joined by hyphens or underscores. Accents go, ß, æ and ø are spelled out, and letters with no Latin form are listed.

Web

UTM Builder

Build a campaign URL with utm_source, utm_medium, utm_campaign, utm_id, utm_term and utm_content. Source, medium and campaign are required.

Web

Open Graph Generator

Write Open Graph meta tags for link previews: og:title, og:type, og:url and og:image, plus description, site name and image alt, with a required-tag check.

Web

Twitter Card Generator

Make twitter:card, site, creator, title, description and image meta tags for a summary or large-image card on X, with checks on @handles and the image URL.

Web

Meta Tag Generator

Generate HTML meta tags for a page: title, meta description, charset, viewport, robots and canonical, plus optional author, keywords and social tags.

Web

Sitemap Generator

Paste one URL per line to get a sitemap.xml: addresses percent-encoded and XML-escaped, duplicates removed, and a lastmod date where you add one after a |.