Answer crawlers, shown first
These fetch pages to answer questions and build search indexes, and they cite you. Blocking one keeps you out of its answers.
Free tool
See which AI crawlers your robots.txt lets in
Enter your domain. We read your robots.txt, match it against each AI crawler's own token, and show which are allowed, which are blocked, and the rule that decided it.
- Enter your domain
- We read robots.txt
- Matched per crawler
- Allowed or blocked
Every AI crawler, sorted by what it does
Training crawlers, kept separate
These collect text to train models and never send a visitor. They use their own tokens, so you can decide about them on their own.
The rule behind every verdict
Each result names the line in your file that decided it, and an unreadable robots.txt is marked not measured, never counted as allowed.
How this works
We fetch /robots.txt from your domain once and apply its groups to each crawler's user agent token, the way the crawler itself does. A file that errors or does not answer is reported as not measured.
Frequently asked questions
Which AI crawlers does the robots.txt check test?
Every AI crawler on the roster we keep for our own sites, from OpenAI, Anthropic, Google, Meta, Perplexity, Apple, Microsoft and others. Your robots.txt is matched against each crawler's own user agent token, and the rule that decided it is shown beside the verdict.
What is the difference between an answer crawler and a training crawler?
An answer crawler fetches pages to answer a question someone is asking now, or to build a search index, and it cites and links back. A training crawler collects text to train a model and never sends a visitor. They use different tokens - OAI-SearchBot and GPTBot are both OpenAI's - so a site can welcome one and turn away the other.
Why does a crawler show as not measured?
Because we could not read your robots.txt: the server errored or did not answer. A crawler that meets an unreachable robots.txt should assume it is shut out, and we never report a file we did not see as permission.
Should I block AI training crawlers?
That is a business decision, and robots.txt lets you make it crawler by crawler. Blocking an answer crawler, though, keeps you out of the answers it gives, so check that group first.