Can AI crawlers read your website?
Enter a domain and see your live robots.txt resolved bot by bot, with the rule that decides each answer.
What this tool does
A robots.txt checker reads the robots.txt file of a website and works out, for every AI crawler, whether it is allowed in. This one fetches your live file, resolves the group that applies to each of the 13 AI crawlers we track (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, CCBot, Bytespider, Applebot-Extended and Amazonbot) using the longest matching user agent, and names the exact line that allows or blocks it. It also lists the sitemaps you declare and any Content-Signal you publish. No account and no email.
Frequently asked questions
It is a tool that reads the robots.txt of a website and resolves, for each AI crawler, whether that crawler is allowed to read the site. Reading the file by eye is unreliable because the rule that applies to a bot is the group whose user agent matches its name most closely, not the first group in the file.
Keep the answer crawlers open, because they are the ones that put you inside a cited answer: OAI-SearchBot, ChatGPT-User, Claude-User, Claude-SearchBot, PerplexityBot, Perplexity-User and Google-Extended. Training crawlers such as GPTBot, ClaudeBot, CCBot or Bytespider are a separate decision and blocking them does not remove you from answers.
No. robots.txt is a sign, not a door. It is obeyed by whoever chooses to obey it, and a scraper that ignores it will read your pages anyway. What actually closes a site is a login, bot rules at the CDN edge and rate limits by IP. Treat robots.txt as a statement of preference.
This is the trap that catches most sites. A group with a named user agent replaces the star group for that bot instead of adding to it, so any rule you write under the star group is invisible to every bot that has its own group. If you add a Disallow later, repeat it inside each named group or those bots will never see it.
Content-Signal is a line that declares how AI companies may use your content, with values such as search, ai-input and ai-train. It lets you stay visible in search and in cited answers while declaring that you do not allow training. It declares a preference and does not enforce it.
Every time you change hosting, install a security plugin or turn on a bot protection setting at your CDN. Most of the blocks we find were never written by hand: they arrived with a default. This tool reads your live file every time, with no cache, so a re-check right after a fix tells you the truth of that moment.