table of contents
Put a validated /robots.txt file at your site root, declare your sitemap, and test every change before publishing. The file must sit at the top-level path, use UTF-8 text, and follow the syntax rules crawlers actually parse. Treat robots.txt as a tool for managing crawl efficiency, never as a way to hide sensitive content, and confirm every change in Search Console or Bing Webmaster Tools before it goes live.
TL;DR:
- If hosting multiple protocols or subdomains, each requires its own robots.txt file to ensure proper crawl management.
- Wildcard patterns like asterisks should be tested carefully to prevent unintended blocking of site content.
- Blocking a directory or parameter with robots.txt does not prevent search engines from indexing the URL if it is linked elsewhere.
- Always verify robots.txt accessibility and correct serving as text/plain with UTF-8 encoding after any deployment or hosting change.
- Regularly review and update robots.txt rules during site updates or redesigns to prevent stale or incorrect instructions from affecting crawl behavior.
What robots.txt is and where to put it
Robots.txt is a plain text file that tells crawlers which parts of your site they can request. It only works when placed at the top-level path of your domain, meaning it must live at yourdomain.com/robots.txt and nowhere else. A copy at /pages/robots.txt or in a subfolder has no effect, and crawlers won’t look for it there.
The protocol scope matters too. A robots.txt file placed at https://yourdomain.com/robots.txt governs only that scheme and host. If your site serves both http and https versions, or runs on multiple subdomains, each combination needs its own robots.txt at its own root, since crawlers treat them as separate authorities under RFC 9309.
File serving requirements are specific. The file must be UTF-8 encoded and served with the Internet Media Type text/plain, according to RFC 9309. A misconfigured server that serves robots.txt as HTML or with the wrong content-type header can cause crawlers to misread or ignore the file entirely.
Redirects add another layer of complexity. If your robots.txt URL redirects, crawlers generally follow a reasonable number of hops to find the actual file, but a long or looping redirect chain risks the file being treated as unreachable. When a robots.txt file is unreachable for an extended period, some crawlers assume no rules exist and proceed to crawl freely, while others may pull back entirely. Neither outcome is something you want to leave to chance, which is why checking server response codes after any hosting or CDN change belongs in a standard release process. For readers managing implementation details like this, Netco Design’s robots.txt best practices resource walks through common deployment pitfalls in more depth.
Core syntax and directives you need to get right
Every robots.txt file is built from a small set of directives, but the way they interact determines whether your rules do what you intend.
User-agent lines group directives by crawler. Product tokens like Googlebot or Bingbot are matched case-insensitively, so Googlebot and googlebot are treated the same way. Each group of directives applies to the user-agent (or agents) named immediately above it, and a general User-agent: * group catches any crawler not named specifically elsewhere.
Allow and Disallow directives control access within a group. When rules conflict, the most specific matching path wins, regardless of whether it’s an Allow or a Disallow. A Disallow: /blog/ paired with Allow: /blog/2026-guide/ permits crawling of that one page even though the broader folder is blocked, because the longer, more specific path takes precedence under RFC 9309 parsing rules.
Special characters extend what these directives can match:
- The asterisk () matches any sequence of characters, useful for blocking patterns like /search or /*?sessionid=.
- The dollar sign ($) anchors a rule to the end of a URL, so Disallow: /*.pdf$ blocks PDF files specifically rather than any URL containing “.pdf”.
- The hash symbol (#) starts a comment, letting you annotate rules for future editors without affecting parsing.
Wildcards are powerful but easy to misuse. A single overly broad asterisk can block far more of a site than intended, so every wildcard rule deserves a test run in a validator before it goes live.
Finally, add a Sitemap: line pointing to your sitemap’s full URL. Sitemap declarations can sit anywhere in the file and don’t need to be grouped under a specific user-agent, since RFC guidance treats them as a separate, standalone record that compliant crawlers read independently. If your sitemap structure itself needs work, Netco Design’s guide to website site map examples covers common formats and organization patterns.
RFC 9309: the standard behind the rules
For years, robots.txt behavior was governed by informal convention rather than a formal specification. That changed when RFC 9309 codified the Robots Exclusion Protocol as an official standard, giving crawler developers and site owners a shared reference for how the file should be parsed and honored.
The standard confirms several requirements worth internalizing. The file must be UTF-8 encoded and served as text/plain, and parsers are expected to handle special characters and apply the most-specific-match rule consistently. It also sets a parsing limit: compliant crawlers should read at least 500 KiB of the file, so an unusually large robots.txt risks having its later rules ignored entirely.
Caching guidance is practical rather than rigid. Crawlers may cache a fetched robots.txt file, and RFC 9309 suggests not relying on a cached copy for longer than 24 hours unless the file is unreachable, in which case cached rules may be used longer. Redirect handling follows a similar logic: crawlers are expected to follow a reasonable chain, generally understood as up to five redirects, before treating the file as unavailable.
Security considerations get direct attention in the standard, and Google’s own guidance echoes the point: robots.txt is not an access control mechanism. It’s a public file, and listing a sensitive directory in a Disallow rule doesn’t hide that directory. It publishes its existence to anyone who reads the file, which can make it more discoverable, not less. Genuine protection for confidential content requires password authentication or server-side access controls, never a crawling rule.
Practical use cases: what to block and when
Most robots.txt problems come from either blocking too little of the low-value crawl surface or blocking too much of what search engines need to render and understand pages.
Common candidates for disallowing include:
- Internal search result pages, which generate near-infinite URL variations with little unique value.
- Faceted navigation parameters (color, size, sort order) that multiply URLs without adding new content.
- Session ID parameters and tracking strings appended to URLs, which create duplicate paths to the same page.
- Temporary or staging directories that shouldn’t be indexed but still get crawled if left open.
Bing’s own guidance for creating a robots.txt file includes examples for blocking internal search paths and directories like /cgi-bin/, and notes that adding bot-specific sections can complicate maintenance if you’re not careful to repeat your general directives across each group.
On the other side of the ledger, avoid blocking CSS, JavaScript, and image files that affect how a page renders. Google has been clear that blocking render-critical resources can prevent crawlers from understanding page structure and content, which can hurt indexing and ranking rather than help it. A page that looks broken to a crawler because its stylesheet is blocked won’t be evaluated the way you intend.
Pro Tip: Before disallowing any parameter pattern site-wide, check your server logs for how much crawl activity it actually generates. Blocking a path that gets ten requests a month wastes effort you could spend elsewhere.
For multi-host setups, whether that’s separate subdomains for a blog, a store, or a staging environment, each host needs its own robots.txt tailored to its content, since rules don’t carry over between them.
Robots.txt, crawling versus indexing, and AI crawler behavior
One of the most persistent misunderstandings about robots.txt is that disallowing a page keeps it out of search results. It doesn’t, necessarily. Google’s own introduction to robots.txt states plainly that the file manages crawler traffic and isn’t a mechanism for keeping URLs out of search results. A blocked page can still appear in search if other sites link to it, since Google may index the URL without ever crawling its content. If your goal is to prevent indexing, use a noindex meta tag or header, or restrict access with authentication, rather than relying on Disallow alone.
AI and LLM crawlers add a newer wrinkle. Crawlers like GPTBot generally check robots.txt and publish their user-agent strings, but real-world behavior varies. Reports from the OpenAI developer community describe site owners dealing with aggressive crawl volume from GPTBot, and in some cases, contacting OpenAI directly and adding an explicit User-agent: GPTBot block resolved the issue, though results weren’t uniform across every site that tried it.
If you want to keep AI crawlers out of specific paths, or your entire site, add a named User-agent block rather than assuming the general wildcard group covers it. When a crawler seems to ignore your rules entirely, check server logs to confirm the exact user-agent string being used, verify the file is actually being served correctly, and escalate to the crawler operator or apply firewall-level rate limiting if the behavior doesn’t stop. Readers building a broader AI visibility strategy can find more context in Netco Design’s AI SEO playbook.
Testing, deployment, and monitoring your robots.txt
A robots.txt file that looks correct on screen can still fail in production, which is why testing belongs in your deployment process rather than after the fact.
- Draft and test changes in a staging environment before touching the live file, catching syntax errors before they affect real crawl behavior.
- Validate the file in Google Search Console’s robots.txt testing tool and in Bing Webmaster Tools, both of which flag syntax issues and show how specific URLs are evaluated.
- Confirm the live file returns an HTTP 200 status and the correct text/plain content-type, since a server misconfiguration can silently break parsing even when the file’s contents are correct.
- Check server logs in the days following any change to confirm crawlers are respecting the new rules and crawl patterns shift as expected.
- Keep a rollback copy of the previous robots.txt version, and set up basic alerting for sudden crawl drops that might signal an accidental block or a CDN caching an outdated version of the file.
Treating this as a checklist rather than a one-time task catches the kind of regression that’s easy to introduce with a single misplaced character.
Common mistakes and how to fix them
The most damaging robots.txt error is an accidental site-wide block, typically a stray Disallow: / left in from a staging environment that gets pushed to production. If organic traffic drops suddenly after a deployment, check robots.txt first. Recovery is usually fast: remove the blanket rule, resubmit the file in Search Console, and request re-crawling of affected pages.
Crawl-delay is another frequent point of confusion. It isn’t part of the current standard and major crawlers handle it inconsistently, so relying on it to manage server load is unreliable at best. Server-side rate limiting or adjusting crawl settings directly in Search Console and Bing Webmaster Tools gives more predictable results.
When a bot appears to ignore your rules entirely, work through it methodically: confirm the file is actually being served with the right status code and content-type, examine the exact user-agent string in your logs rather than assuming it matches the name you blocked, and contact the crawler’s operator if the behavior persists after you’ve ruled out a configuration error on your end.
Quick checklist and ready-to-adopt templates
Before publishing any robots.txt update, run through this list:
- File sits at /robots.txt on the correct host and protocol.
- Encoding is UTF-8 and the server returns text/plain.
- A Sitemap: line points to your current sitemap URL.
- Rules have been tested in Search Console or Bing Webmaster Tools.
- Server logs are being monitored for at least a few days post-change.
A few generic patterns cover most needs. A default setup allowing full access while declaring a sitemap looks like User-agent: * followed by Disallow: (left blank) and a Sitemap: line. Blocking a staging subdomain entirely uses User-agent: * with Disallow: /. Blocking internal search pairs User-agent: * with Disallow: /search. Blocking a specific crawler by name uses its own User-agent: line, such as User-agent: GPTBot, followed by Disallow: /.
Netco Design’s perspective on robots.txt policy and maintenance
Robots.txt deserves less mystique and more routine. It’s a crawl-management tool, not a security layer, and treating it as anything more than that is where most of the real damage happens. The teams that get the most value from their robots.txt file are the ones who check it the same way they check a deployment log: as a standard step, not a special occasion.
Our recommendation is to fold a robots.txt review into every site release and periodic technical audit, checking not just syntax but whether the rules still match current site structure. New sections, retired staging environments, and changed URL patterns all leave stale rules behind if nobody’s watching.
When a crawler, particularly a newer AI agent, seems to ignore your directives, don’t spend weeks guessing. Confirm the basics are correct, then escalate to the vendor or apply a firewall rule if the crawl volume is causing real strain on your servers.
— Lori Piencykoski
Get a robots.txt and technical SEO audit from Netco Design
A correct robots.txt file is a small piece of a much larger technical SEO picture, and most site owners don’t have the time to audit crawl directives, sitemap structure, and indexing settings on top of running their business. That’s the gap our team closes. Netco Design’s SEO services include a full technical review, robots.txt included, alongside the broader crawl and indexing setup that determines whether search engines and AI crawlers can find and understand your site the way you intend.
We build sites mobile-first and structure them for both traditional search and AI-driven discovery, so a robots.txt fix doesn’t happen in isolation from everything else affecting your visibility. If you’re not sure whether your current file is helping or quietly working against you, request a technical SEO review and we’ll walk through what we find.
Sources
For deeper reading, RFC 9309 is the specification itself and the final word on parsing behavior. Google Search Central’s robots.txt introduction covers crawling versus indexing in plain terms, while Bing Webmaster Tools offers practical setup examples. For AI crawler questions, OpenAI’s developer community threads document real GPTBot behavior and how site owners have resolved conflicts directly with the vendor.
- Introduction to robots.txt — Google Search Central
- How to create a robots.txt file — Bing Webmaster Tools
- GPTBot makes over 10000 request to my website — OpenAI Developer Community
- RFC 9309 (robots.txt details) — RFC Editor
FAQ
What is the difference between disallow and noindex?
Disallow in robots.txt tells crawlers not to request a page, but it doesn’t guarantee the page stays out of search results if other sites link to it. Noindex is a meta tag or HTTP header instruction that tells search engines not to show a page in results even if they crawl it, and it’s the correct choice when the goal is keeping a specific page out of search entirely.
Can robots.txt block AI crawlers like GPTBot?
Yes, adding a named User-agent block such as User-agent: GPTBot followed by Disallow: / is the standard way to signal that a crawler shouldn’t access your site. Compliance varies by crawler, and the OpenAI developer community has documented cases where contacting the vendor directly was needed alongside the robots.txt rule.
Does blocking a page in robots.txt remove it from Google?
No, robots.txt controls crawling, not indexing, and Google has stated that a blocked page can still appear in search results if it’s linked from elsewhere. To remove a page from search results, use a noindex tag or restrict access with authentication instead.
How often should I update my robots.txt file?
There’s no fixed schedule, but reviewing it during major site changes, redesigns, and periodic technical audits catches rules that no longer match your current structure. Checking server logs after any update confirms crawlers are responding to the new rules as expected.
Is robots.txt a security measure?
No, robots.txt is a publicly readable file that tells crawlers what to avoid, but it doesn’t restrict actual access to any page or file. For genuinely sensitive content, use password protection or server-side access controls rather than relying on a Disallow rule.
