Check a URL against every documented crawler at once, and see which rule decided it.
A full URL or a path beginning with /
The crawler token, or * for the default group
Paste a robots.txt and enter a path to see whether it may be crawled, and which rule decides.
Being allowed to crawl is not the same as being indexed. A permitted URL may still be left out of an index for other reasons, and a blocked URL can still appear in results — without a snippet — if other sites link to it. To keep a page out of an index, allow crawling and use a noindex tag.
A blocked path
See the rule responsible
/admin/settingsAn exception inside a blocked folder
Longest match wins
/folder/public/pageA named crawler
Ignores the * group entirely
GooglebotThe whole file, exactly as served.
A path beginning with /, and a user-agent token like Googlebot.
Allowed or blocked, with the exact line and pattern that decided it.
Knowing a URL is blocked is half an answer. The tool names the line and pattern responsible, which is what you need to fix it.
Longest-match-wins, Allow-on-tie and prefix-based agent matching are all per specification, so the result matches what a crawler would do.
A named group silently overriding the wildcard group is a common cause of confusion, and it is made visible.
Testing one agent answers the question you thought to ask. Running all of them finds the search crawler you blocked without meaning to, and names the training tokens your file is still missing.
Blocking the search bot while allowing the training bot is the exact inverse of what people intend, and it is invisible unless you check both. The audit calls it out by name.
Paste a robots.txt file and a path, and find out whether that path may be crawled — and crucially, which line made the decision. The answer often surprises people, because precedence is not what intuition suggests: the longest matching pattern wins regardless of where it appears in the file, an Allow only beats a Disallow on an exact length tie, and a crawler matched by name ignores the wildcard group entirely. All of that is implemented per RFC 9309 and the deciding rule is shown with its line number. It also runs the path against all 24 catalogued tokens from nine operators, which catches the mistakes a one-agent-at-a-time check cannot: a search crawler blocked by accident, an AI opt-out copied from an article that predates the tokens being split, and the inverted pair where a site blocks the search bot while leaving the training bot allowed.
Keep going
Related pages in SEO Tools, plus what others are using right now.
More seo tools that pair well with this one.
Build a valid robots.txt, with platform presets and per-crawler AI controls.
Measure a title and description by rendered width, with a live preview of the result.
Produce a canonical link tag and check the URL for the mistakes that break it.
Build a valid XML sitemap from a list of URLs, with correct escaping and dates.
Check a sitemap against the protocol — structure, namespace, dates and limits.
Word count, reading time, sentence length and readability for a piece of writing.
Create strong random passwords with full control over length and characters.
Format, validate and minify JSON, with errors that point to the exact line.
Work out any percentage — of a number, as a share, or as a change.
Count words, characters, sentences and paragraphs as you type.
Convert between length, weight, temperature, volume, speed, data and more.
Count characters with and without spaces, against the limits that matter.
Find the mean, median, mode and range of a list of numbers, with the working shown.
Sample and population standard deviation and variance, both shown, with the working.
Turn a logo into favicon.ico, the PNG sizes a site needs, and the HTML to declare them.
Check a downloaded file against its published checksum, without uploading anything.
Stamp text like DRAFT or CONFIDENTIAL across a PDF, in your browser.
Turn JSON into readable YAML, with quoting and multiline strings handled properly.