A robots.txt file tells well-behaved crawlers which pages they may request. It does not control what an AI company does with content it has already collected. Google’s Gary Illyes said so in a video published on October 21, 2026. If you want to block a specific AI bot, you must name that bot in the file. If you need real protection, use a login, not robots.txt.
What changed
Google Search Central released the second episode of its Non-Canonical Cabinet video series, with Gary Illyes and Cherry Prommawin. According to Search Engine Journal, Illyes explained that robots.txt is an access preference. It “doesn’t dictate what a system does with the data after it has already crawled it.” Search Engine Roundtable summed it up this way: blocking an AI crawler from a page is different from telling a company not to train on your content.
The video also made three practical points. Google’s parser keeps only a few directives and silently discards lines it does not understand, so a rule that looks clear to a human may do nothing. To block one AI crawler you must target its specific user agent token, because an invented directive is ignored. And robots.txt is not a security tool, since it only asks compliant crawlers to stay out. PPC Land added that Google-Agent, a fetcher that acts on a user’s request, does not follow robots.txt.
What robots.txt can and cannot do
| What you want | Does robots.txt do it? | What to use instead or as well |
|---|---|---|
| Ask a named crawler to skip certain URLs | Yes, if the bot is well behaved and you name its token | Test the file, and check each AI company’s documentation |
| Stop AI from using content already collected | No | Contracts, licensing terms and each company’s opt-out tools |
| Keep private pages secret | No, the file is public | Login, password or server-level access rules |
| Limit Google’s use of your content for Gemini training and grounding | Only through the Google-Extended token, per Search Engine Journal | Google’s product documentation |
Source: Google Search Central video, as reported by Search Engine Roundtable (Barry Schwartz, Oct 21), PPC Land (Luís Rijo, Oct 21) and Search Engine Journal (Matt G. Southern, Oct 22, 2026).
Why this matters beyond the rules
Even a correct rule is not a guarantee. PPC Land cited TollBit data for the first half of 2026 showing that about 15% of identified AI page-fetching agents reached URLs that European sites had disallowed. It also cited a HasData check from July in which 234 of 592 sites that disallowed GPTBot still served it a live page. Treat these as indicators from third parties, not as proof about your own site.
There is also no agreed standard yet for saying “crawl this, but do not train on it.” Search Engine Journal reports that an IETF draft to add usage preferences to robots.txt is still a working group document. For a business owner, the lesson is simple: robots.txt is one lever, and it does not cover every risk.
What to do about it
- Decide what you want first. Being cited by AI search tools can bring buyers, while pure training bots send none. Choose a position for each bot, using the comparison in our SEO vs AEO vs GEO guide.
- Name the bots you want to block. Add the exact user agent token for each one, followed by a Disallow rule, and never block Googlebot by accident.
- Test your file. Unknown lines are dropped without an error, so use a robots.txt tester and check that important pages stay crawlable.
- Protect private content properly. Put customer areas, drafts and price lists behind a login. Do not rely on a Disallow line to hide them.
- Review it every quarter. New AI crawlers appear often. Our AEO and technical SEO services include a crawler access review, and our earlier post on Meta’s new crawling terms shows how quickly the rules change.
The bottom line
Robots.txt still matters for how search engines and AI search tools find your pages, but it is not a privacy or copyright shield. Set clear rules for the bots you care about, protect what is truly private, and see how this fits a wider visibility plan at GCL Marketing. You can also browse more articles by topic.
Frequently Asked Questions
Does blocking a bot in robots.txt stop AI from using my content?
Not fully. Google’s Gary Illyes said robots.txt does not dictate what a system does with data after it has already crawled it. It only tells compliant crawlers which URLs they may request. Content collected earlier, or by a bot that ignores the rule, can still be used.
How do I block one specific AI crawler?
Name that crawler’s user agent token in your robots.txt file, followed by a Disallow rule. Google explained that a made-up or generic directive is simply ignored. Check each AI company’s documentation for its exact token, then test the file, because lines the parser does not understand are discarded silently.
Will blocking AI crawlers hurt my Google rankings?
Blocking a non-Google AI crawler does not affect Google Search. According to Search Engine Journal, the Google-Extended token covers Gemini training and grounding and does not affect Search inclusion or rankings. Blocking Googlebot itself would remove your pages from Search, so check every rule before you publish the file.
Is robots.txt a security tool for private pages?
No. Search Engine Roundtable quoted the video as saying robots.txt is not a security mechanism. It politely asks compliant crawlers to stay out, and the file is public, so anyone can read the paths you list. Keep anything private behind a login or server-level access rules.
Should my business block AI crawlers at all?
It depends on your goals. Blocking can limit training use, but it can also reduce how often AI search tools find and cite you. Most local, service and e-commerce businesses want to be cited, so decide bot by bot. Allow bots that send visitors, and consider blocking pure training bots.