Free SEO tool

Robots.txt Generator and Tester

Build a robots.txt file with the AI bot presets already filled in, then test any URL against it before you upload.

A robots.txt generator writes the plain text file that sits at /robots.txt and tells crawlers which URLs they may fetch. This one also groups the AI bots by what blocking them costs you, writes the Shopify Liquid version, and tests the result the way Google's own parser would.

It runs entirely in your browser. Nothing you type is sent anywhere and there is no sign-up.

Every user agent token was checked against the operator's own documentation in September 2026.

Build and test your robots.txt file

What is a robots.txt generator?

A robots.txt generator is a form that writes the file for you, so you do not have to remember the syntax or which crawler answers to which name.

Most of them were written before AI crawlers existed.

They hand you a file full of search engine rules and a Crawl-delay line that Google ignores, and they say nothing about the decision that actually matters in 2026: which AI bots you let through.

This one starts there.

Blocking a training crawler keeps you in AI answers. Blocking an answer crawler takes you out of them.

Those are different decisions, and most tools treat them as the same one.

Below the tool I cover how the rules are actually applied, which AI bots I block for clients and which I never do, what Shopify does differently, and how to fix the "Blocked by robots.txt" warning in Search Console.

How this robots.txt generator works

You pick which crawlers to allow and which to block, add any paths you want kept out, and the tool writes a valid file in the order crawlers expect to read it.

Three things happen behind that.

It groups rules by user agent, not by bot. Block GPTBot, ClaudeBot and CCBot and you get one block: three User-agent lines above a single Disallow: /. That is how the specification is written, and it keeps the file short enough to read.

Files that repeat the same rule twenty times are where mistakes hide.

It writes the wildcard group last. A crawler uses the most specific group that matches its name and ignores every other group, so the order does not technically matter.

It matters to you, six months from now, when you are trying to work out why a page dropped out.

It refuses to write rules that do nothing. Google does not support Crawl-delay, Noindex or Host in robots.txt, so a crawl delay is written only into the groups for crawlers that actually read it.

A bot you mark as allowed gets a comment rather than a group of its own. Why? Because naming a crawler in a group makes it ignore User-agent: * entirely, so an allow group either repeats the wildcard rules word for word or, far worse, quietly exempts that bot from all of them.

Most generators write it anyway.

How the rules are applied when a crawler reads your file

This is the part that generates support tickets, so the tester follows Google's open-source parser rather than doing a simple string match.

Step one: pick the group. A crawler reads each User-agent value up to the first character that is not a letter, hyphen or underscore, so Googlebot/2.1 counts as Googlebot, and uses the group whose token exactly matches its own name, case insensitively.

Googlebot obeys a group headed User-agent: Googlebot and completely ignores User-agent: * in the same file.

I review robots.txt on every technical audit, and this is the mistake I flag most often.

Somebody adds a specific group for one crawler, and that crawler silently stops obeying every rule in the wildcard group. Bing has published the same warning about Bingbot.

If the same token appears in two separate groups, Google merges them, and so does the tester.

Step two: match the path. Paths are case sensitive, even though user agent names are not. * matches any run of characters and $ anchors to the end of the URL. Everything else is a literal prefix match, so Disallow: /news also blocks /newsletter.

Step three: the longest rule wins. Where an Allow and a Disallow both match, Google uses the more specific one, measured by the length of the path pattern, and a trailing $ counts towards that length.

If they are the same length, the Allow wins.

That is how Disallow: /search/ plus Allow: /search/about blocks the folder but keeps one page in it.

Which AI bots should you block in robots.txt?

There is no single right answer. However, there is a right way to frame it: split the bots by what they do, then decide once per group.

Training crawlers take your content to train a model. GPTBot, ClaudeBot, Google-Extended, CCBot and Applebot-Extended sit here.

Blocking them is a licensing decision. You get no traffic from training, and you are not removed from anything by opting out.

Google is explicit that Google-Extended "does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search", and Apple says pages that disallow Applebot-Extended "can still be included in search results".

Answer crawlers build the index behind AI answers. OAI-SearchBot, Claude-SearchBot and PerplexityBot sit here.

Blocking these takes you out of the results those products cite. That is an answer engine optimisation decision, not a privacy one.

If you have read anything I have written about AI visibility, you will know I think this is the group most sites get wrong. They block everything with "AI" in the name and then wonder why they never appear in ChatGPT.

User-initiated fetchers visit because a person asked. ChatGPT-User, Claude-User and Perplexity-User sit here.

These are closer to a browser than a crawler, and the operators say so. Perplexity states that Perplexity-User "generally ignores robots.txt" because the user requested the fetch, and Meta says the same of meta-externalfetcher.

Blocking them in robots.txt is a request, not a control.

My default for a business that wants to be found? Block the training crawlers, allow the answer crawlers, and leave the fetchers alone. That is the second preset in the tool.

Robots.txt examples you can load into the tester

Each of these is a pattern I have either written for a client or found broken on an audit. Load one, change the URL or the bot, and watch which rule wins.

User-agent: *
Disallow:

Sitemap: https://www.example.com/sitemap.xml

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: Bytespider
User-agent: meta-externalagent
Disallow: /

User-agent: *
Disallow:

Sitemap: https://www.example.com/sitemap.xml

User-agent: *
Disallow: /search/
Allow: /search/about

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /?s=
Disallow: /search/

Sitemap: https://www.example.com/sitemap_index.xml

User-agent: *
Disallow: /

User-agent: *
Disallow: /cart
Disallow: /checkout

User-agent: Googlebot
Disallow: /internal/

Fixing "Blocked by robots.txt" in Search Console

Search Console has two warnings with almost the same name, and they need opposite fixes.

Blocked by robots.txt means Google wanted to crawl the URL and your file said no.

If the page should be in search, paste your robots.txt into the tester above, enter the URL, pick Googlebot, and the tester names the exact line that blocked it. Fix that line, then request indexing through URL Inspection.

However, if the page should not be in search, the warning is working as intended and you can leave it.

Indexed, though blocked by robots.txt is the awkward one. Google found links to the URL, could not crawl it, and indexed the address anyway with no snippet.

Robots.txt cannot fix this, because the fix is a noindex tag and Google cannot see a noindex on a page it is not allowed to crawl.

Remove the disallow, let Google crawl the page and read the noindex, and the listing drops out. Once it has gone, you can put the disallow back if you want to save the crawl budget.

The order matters. Do it the other way round and the URL stays indexed indefinitely.

Robots.txt on Shopify: why you cannot just upload a file

Shopify serves robots.txt from a Liquid template, not from a file in the root, so there is nothing to upload. You create templates/robots.txt.liquid in the theme editor and Shopify renders it at /robots.txt.

The trap is that it is very easy to replace the whole thing with plain text.

Shopify's default groups block the cart, the checkout, the account pages and the sorting and filtering parameters that generate near-infinite URLs on a collection page.

Throw those away and you hand a crawler an unbounded set of URLs on a site that is usually already short of crawl budget. I go through where those URLs come from in Shopify duplicate content.

Shopify also updates those defaults, most recently to cover agent-driven checkouts, and a hardcoded file never picks that up.

So the Liquid this tool writes loops over robots.default_groups and appends your rules inside each one. Switch the platform to Shopify above and the block appears under the plain text output.

If you want the wider picture on Shopify crawl control, start with the Shopify SEO checklist.

What this tool will not do

Worth being straight about the limits before you rely on it.

• It does not fetch your site. The tester works on text you paste. That is deliberate, because it means you can check a file before you publish it, but it also means the tool cannot tell you what is live right now. For that, use the robots.txt report in Search Console, or run the crawl step of my technical SEO audit checklist.
• It does not stop a crawler. Nothing in a robots.txt file does. Bots that ignore the standard are blocked at the server or the CDN, not here. Independent log analyses have repeatedly caught Bytespider fetching disallowed URLs, and I have flagged it in the tool for that reason.
• It does not deindex anything. See the Search Console section above. Blocking a URL that is already indexed can actually preserve the listing, because the crawler can no longer see the noindex you added.
• It does not know about bots that launched this week. The list was verified in September 2026. New agents appear constantly, which is why there is a field to add your own.
• It does not handle llms.txt. That is a separate proposal and a separate file, and it is not a replacement for robots.txt. I have written up what llms.txt does on Shopify separately.

Frequently asked questions

Is there a robots.txt for AI?

There is no separate file. AI crawlers read the same /robots.txt as search engines and obey it the same way, using their own user agent tokens such as GPTBot or ClaudeBot. What has changed is that the same file now controls two different things at once: whether you are crawled for search, and whether you are used for AI training and AI answers. You separate them with per-bot groups, which is what this tool builds.

Does blocking GPTBot remove me from ChatGPT?

No. GPTBot is the training crawler. The bot that builds ChatGPT's search index is OAI-SearchBot, and OpenAI documents them as separately controlled. Block GPTBot and you opt out of training while staying eligible to appear in ChatGPT's search results. Block OAI-SearchBot and you leave those results. I cover what gets a page cited there in how to rank in ChatGPT.

Does blocking Google-Extended remove me from AI Overviews?

No. Google-Extended controls whether your content trains Gemini and grounds Gemini Apps and Vertex AI. AI Overviews and AI Mode are part of Google Search, and Google says there are "no additional requirements to appear in AI Overviews or AI Mode" beyond the normal Search controls. To limit what appears there, the levers are nosnippet, data-nosnippet, max-snippet and noindex, not Google-Extended. Blocking Google-Extended and expecting to leave AI Overviews is a misunderstanding I see a lot, and an expensive one. If AI Overviews are the goal, this is what actually moves them.

Does robots.txt still work?

For crawling, yes. Every major search engine and every major AI operator honours it, and the standard was formalised as RFC 9309 in 2022. For keeping pages out of the index it has never worked, and that has not changed. It controls crawling, not indexing.

Is robots.txt legal?

It is a technical convention, not a legal instrument. RFC 9309 made it a formal standard, but a standard is not a law, and the file itself grants and withholds nothing in a legal sense. A crawler that ignores it is breaking a convention. Whether that is also a breach of your terms or of copyright is a question for a lawyer, and for the courts currently hearing exactly those cases. What I can say is that a Disallow on the training crawlers is the accepted way to signal that you do not license your content for AI training, and the major operators state that they honour it.

Should the sitemap go in robots.txt?

Yes, and it is the one line that costs nothing. A Sitemap: line with the absolute URL lets every crawler find your sitemap without you submitting it to each one separately. It is not tied to any User-agent group, so it goes at the bottom, and you can list more than one. A sitemap on a different host is allowed here too. Robots.txt is the documented way to point at one.

Where does the robots.txt file go?

The root of each host and protocol, at https://example.com/robots.txt. It does not apply to subdomains, so shop.example.com needs its own. On Shopify it is a theme template rather than a file. On WordPress, a physical file in the web root overrides the virtual one WordPress generates.

Why does my robots.txt say Crawl-delay but Google ignores it?

Because Google never supported the field. Its documentation lists the fields it reads and notes that "other fields such as crawl-delay aren't supported". Yandex stopped reading it on 22 February 2018. Bing is the one major engine that still does. If Googlebot is crawling too hard, the control is the crawl rate setting in Search Console, not this file.

How big can a robots.txt file be?

Google enforces a limit of 500 kibibytes and ignores everything past it. You will not get near that with per-bot groups. If you are close, the real problem is that you are managing URL patterns in robots.txt that belong in canonical tags or parameter handling.

Should I block AI bots at all?

If your business model is selling access to your content, block the training crawlers. If your business model is being found, allow the answer crawlers and think hard before you block anything else. Most of the sites I audit have no commercial reason to block either, and a few have quietly blocked themselves out of AI answers without realising it. That is worth checking before you change anything. My technical SEO audit reviews exactly this file, including which AI crawlers it lets through, and the AI visibility audit shows you what the answer engines say once they can get in.

Built by Graeme Whiles, an SEO and AEO consultant. Every user agent token in this tool was verified against the operator's own documentation. Last checked 13 September 2026.