TL;DR Summary:
Crawling Isn’t Content Control: Robots.txt can tell AI crawlers which pages to avoid, but it cannot control how companies use content after crawling it.Only Four Directives Work: Under RFC 9309, compliant crawlers recognize only user-agent, allow, disallow, and sitemap, while scripts, formatting errors, and unsupported lines are ignored.Emojis Cannot Block Bots: User-agent tokens must use letters and dashes, so emojis and non-English characters fail, while international URL paths should be percent-encoded.Robots.txt Isn’t Security: The file relies on crawler cooperation, meaning non-compliant bots can ignore it and access pages anyway.Can you block AI crawlers with emojis in your robots.txt file? Google answered this and several other common questions on October 21, 2026, through a video from Gary Illyes and Chery Prommawin. The video lays out clear robots.txt rules for AI crawlers, international characters, and formatting mistakes that site owners make often, and it matters because many people treat this file as more flexible than it actually is.
What Robots.txt Rules for AI Crawlers Actually Cover
Gary Illyes explained that robots.txt works as an access control file, not a content usage policy. It tells a crawler which pages to avoid. It says nothing about what a company does with content after it has already crawled that page. If you want to stop an AI crawler from training on your content, you need a separate policy or agreement. Robots.txt rules for AI crawlers only control the crawling step, not what happens to your data afterward.
To block a specific AI crawler, you must use its exact recognized user-agent token, the name the crawler identifies itself by. Illyes said a made-up rule like blocking by a nickname or description does nothing. The parser ignores anything it does not recognize and moves on. Since it’s often hard to know just by looking at your robots.txt file whether a rule is actually being honored, platforms like Screpy are useful here, as they monitor AI and bot traffic directly so you can confirm which crawlers are actually respecting your directives versus which ones are slipping through unnoticed.
Robots.txt Follows RFC 9309, and Four Directives Only
Illyes confirmed that Google follows the internet standard RFC 9309 for robots.txt files. The parser reads line by line and only recognizes four directives: user-agent, allow, disallow, and sitemap. Anything outside those four, including broken formatting or extra spacing, gets silently discarded. The crawler simply skips the line and continues.
Cherry Prommawin mentioned finding a robots.txt file that was 97 percent script and only 3 percent actual directives. That script gets ignored by compliant crawlers, so it only wastes bandwidth. You can technically add comments starting with a hash symbol, and crawlers will skip them entirely, but Illyes called this a questionable choice rather than a useful practice.
Emojis and Non-English Characters in User-Agent Tokens Fail
User-agent tokens only accept letters and dashes. Illyes confirmed that emojis and non-English characters cannot work in this field because an emoji is a multibyte UTF-8 character, a format that uses several bytes to represent one symbol, and it does not fit the letter-dash format at all. Any attempt to target a crawler using an emoji as its name fails completely, since the rule never matches a real crawler.
Percent-Encoding Protects International URL Paths
For URL paths containing international characters, such as German umlauts, Chinese characters, or emojis, Google recommends percent-encoding, a method that converts special characters into a standard text format every browser and crawler can read. Googlebot’s parser handles raw UTF-8 characters automatically, according to Illyes, but older or less sophisticated crawlers might not. Using the encoded version in your robots.txt rules keeps the rule readable across different crawler systems. Skipping this step risks having a bot ignore your disallow line entirely and crawl pages you meant to protect.
Robots.txt Is Not a Security Tool
Illyes repeated a point worth remembering: robots.txt requests cooperation from crawlers, it does not enforce access restrictions. A crawler that ignores the file can still access your pages. Compliant crawlers, including Googlebot, follow the rules, but nothing stops a non-compliant bot from disregarding them completely.
Check your robots.txt file for correct syntax and confirm you use exact user-agent tokens for any AI crawler you want to block. Percent-encode any international characters or emojis in your URL paths rather than leaving them raw. These robots.txt rules for AI crawlers only work when the formatting matches what RFC 9309 expects. And since even a perfectly formatted file can’t guarantee compliance, it helps to verify the results: if you want to confirm your robots.txt rules are actually working as intended, tools like Screpy’s AI and bot traffic monitoring can show you exactly which verified AI crawlers are reaching your content and flag URLs that may be silently blocked.
