Build a valid robots.txt, with platform presets and per-crawler AI controls.
Optional. Used to fill in the Sitemap line.
Start from a platform
Everything crawlable, with the sitemap pointed at. This is the right answer far more often than people expect, and an empty Disallow is how you say it explicitly.
Where it goes. Save as robots.txt in the root of your domain. It has no effect anywhere else.
AI and search crawlers
Tick a crawler to disallow it. These are separate decisions on purpose. Blocking a training bot does nothing to your search visibility, and blocking a search bot does nothing to stop training.
This list is published as open data at /crawlers.json — CC0, every token read from the operator’s own documentation.
Rules
One absolute URL per line
Validation of this output
robots.txt controls crawling, not indexing. A disallowed URL can still be listed in search results without a description if other pages link to it — to keep a page out of an index entirely, allow crawling and serve a noindex meta tag. The file must sit at the root of each domain and subdomain to have any effect, and it is a request rather than an enforcement: well-behaved crawlers honour it, and anything scraping you deliberately will not.
Block an admin area
The most common rule
Disallow: /admin/Allow everything
An explicit empty Disallow
Disallow:Block one crawler only
A named group overrides *
User-agent: GPTBotName the crawlers it applies to, or use * for all of them.
Allow and Disallow paths, one per line, starting from the site root.
Save it as robots.txt in your site root — it only works there.
Consecutive User-agent lines are grouped properly, and a rule-free group still gets the empty Disallow it needs to be meaningful.
The output is parsed back through the same RFC 9309 parser that powers the validator, so structural mistakes surface immediately.
Crawl-delay is offered with a note that Google ignores it, rather than implying every crawler obeys everything.
Training, search and user-triggered fetches are separate choices, each labelled with what actually happens when you block it. One checkbox for "block AI" would hide the fact that it costs you AI search visibility you probably wanted to keep.
Shopify will not let you upload one, WordPress serves a virtual file until a real one exists, and Next.js has none until you add a route. Each preset says how to actually apply it.
Generate a robots.txt file that crawlers will actually parse the way you intend. The structure matters more than people expect: consecutive User-agent lines share the rules beneath them, a rule line closes the header, and a group with no rules needs an explicit empty Disallow to mean anything. This builder handles all of that, starts you from a preset for WordPress, Shopify, Next.js or Ghost, and tells you where the file actually goes on each — which is not always somewhere you can upload to. It also splits the AI crawlers by what they are for, because blocking model training and staying visible in AI search are two different decisions that most tools collapse into one checkbox. All 24 tokens are taken from the operator's own documentation, across nine operators, and the one crawler nobody documents is labelled as such rather than quietly listed beside the rest. The output is parsed back through the validator so what you copy is known-good rather than assumed-good.
Keep going
Related pages in SEO Tools, plus what others are using right now.
More seo tools that pair well with this one.
Check a URL against every documented crawler at once, and see which rule decided it.
Measure a title and description by rendered width, with a live preview of the result.
Produce a canonical link tag and check the URL for the mistakes that break it.
Build a valid XML sitemap from a list of URLs, with correct escaping and dates.
Check a sitemap against the protocol — structure, namespace, dates and limits.
Word count, reading time, sentence length and readability for a piece of writing.
Create strong random passwords with full control over length and characters.
Format, validate and minify JSON, with errors that point to the exact line.
Work out any percentage — of a number, as a share, or as a change.
Count words, characters, sentences and paragraphs as you type.
Convert between length, weight, temperature, volume, speed, data and more.
Count characters with and without spaces, against the limits that matter.
Find the mean, median, mode and range of a list of numbers, with the working shown.
Sample and population standard deviation and variance, both shown, with the working.
Turn a logo into favicon.ico, the PNG sizes a site needs, and the HTML to declare them.
Check a downloaded file against its published checksum, without uploading anything.
Stamp text like DRAFT or CONFIDENTIAL across a PDF, in your browser.
Turn JSON into readable YAML, with quoting and multiline strings handled properly.