robots.txt rules: precedence, wildcards, and common mistakes

Most robots.txt files are short, and that is why mistakes slip through. One stray slash or a misplaced group can block a whole section, or the whole site, and nothing warns you. This guide covers how crawlers read the file: which rule wins, how wildcards behave, and which errors are worth auditing for today.

It focuses on Google's documented behavior, since that is the clearest. Other crawlers follow the same Robots Exclusion Protocol, but small details can differ, so check each vendor's docs for the bots you care about.

How crawlers read the file

A robots.txt file is a list of groups. Each group starts with one or more User-agent lines and is followed by Allow and Disallow rules. A crawler picks the single group that matches its name most specifically and ignores the rest.

That last point trips people up. If you write a group for Googlebot and a group for everything else, Googlebot does not also obey the general group. Rules are not inherited. If the general group blocks a folder and you want Googlebot blocked too, repeat the rule in its own group.

• The file must live at the root of the host, like example.com/robots.txt. Each subdomain, protocol, and port needs its own.
• Paths are case-sensitive. Disallow: /Admin does not block /admin.
• Directive names are not case-sensitive, but keep them tidy anyway.
• Google reads only the first 500 KiB. Anything past that is ignored, so keep the file small.

Rule precedence and wildcards

When an Allow and a Disallow both match a URL, Google follows the most specific rule, meaning the one with the longest matching path. If both match with equal length, the less restrictive rule wins, so Allow beats Disallow.

Here is a common pattern. You block a folder but want one file inside it crawled. Disallow: /private/ plus Allow: /private/press-kit.pdf works, because the Allow path is longer. Order in the file does not matter to Google.

Two wildcard characters are supported. The asterisk matches any run of characters. The dollar sign anchors the end of the URL.

• Disallow: /*?sessionid= blocks any URL carrying that parameter.
• Disallow: /*.pdf$ blocks URLs ending in .pdf, but not /file.pdf?download=1.
• Disallow: /shop blocks /shop, /shop/, and /shopping-guide. Add a trailing slash if you only mean the folder.

Prefix matching is the source of many accidents. Always read a rule as 'every path that starts with this'. A short rule like Disallow: /p catches every URL beginning with /p.

Mistakes worth auditing

Blocking when you meant to deindex

Disallow stops crawling. It does not remove a page from the index. A blocked URL can still appear in results if other pages link to it, usually without a description. Worse, a crawler that cannot fetch the page cannot see a noindex tag on it. To deindex, allow the crawl and use noindex or the X-Robots-Tag header.

Putting noindex in robots.txt

Google stopped honoring a noindex line in robots.txt in 2019. If an old file still has one, it does nothing for Google. Move it to a meta tag or header.

Blocking CSS and JavaScript

If Googlebot cannot fetch the files a page needs to render, it may misjudge the layout and content. Unless you have a clear reason, leave scripts and stylesheets crawlable.

Expecting crawl-delay to work everywhere

Google ignores Crawl-delay. Some other crawlers honor it, Bing among them. If Googlebot is straining your server, fix the server or check the Crawl Stats report instead of adding a delay line.

Leaving a staging block in production

A Disallow: / left over from a staging launch is a classic. It is easy to ship during a migration and hard to notice, because the site keeps working for visitors.

Misreading server errors

Google treats a 404 or other 4xx on robots.txt as if no restrictions exist. A 5xx response is treated as a temporary full block, and if it persists, Google may fall back to a cached copy. A flaky robots.txt is a real crawling risk, so make sure it is served reliably and fast.

AI crawlers and group names

The same rules apply when you write groups for AI crawlers such as GPTBot, ClaudeBot, or PerplexityBot. Each needs its own User-agent line, and a wildcard group will not cover a bot that has its own named group. Names change over time, so confirm the current user-agent token in each vendor's documentation before relying on a rule. Also remember that robots.txt is a request, not enforcement. Well-behaved bots honor it. Others may not.

A five-minute audit

1. Open your live robots.txt in a browser and confirm it returns a 200 status.
2. List every group and check that each bot you care about lands in the group you intended.
3. Test your most important URLs against the rules. Search Console has a robots.txt report, and the URL Inspection tool shows whether a URL is blocked.
4. Search the file for Disallow: / , broad prefixes, and any noindex line.
5. Add the Sitemap directive with a full absolute URL if it is missing.

The takeaway: treat robots.txt like code. Keep it short, read every rule as a prefix, repeat shared rules inside each named group, and re-check it after every migration or CMS change.