Robots.txt & XML Sitemap Guide 2026: Rules for Google, AI Search and LLM Crawlers

What robots.txt does

Robots.txt is a plain-text file at the root of your host (for example https://www.example.com/robots.txt) that gives crawl instructions to bots. It follows the Robots Exclusion Protocol (RFC 9309) and is the first file a well-behaved crawler requests.

It controls crawling, not indexing. A blocked URL can still appear in results as a bare link if other sites point to it, and a blocked page can never show a noindex tag because the crawler cannot read it. If you want a page out of search, allow crawling and serve noindex instead. This is the core idea behind our site crawlability and indexation guide.

Robots.txt rules to remember

Example: wildcard and exact-match rules

User-agent: *
Disallow: /cart/
Disallow: /checkout/
Disallow: /my-account/
Disallow: /search/
Disallow: /*?*sort=
Disallow: /*?*sessionid=
Allow: /search/help$

Sitemap: https://www.example.com/sitemap.xml

Common robots.txt errors and fixes

These come from real technical audits of large B2B e-commerce sites, the same checks we run in our SEO audit services.

Robots.txt for LLMs and AI crawlers

AI companies now run separate bots for training, search indexing and user-triggered fetching. Blocking the wrong one can remove you from AI answers or do nothing at all. This is the foundation of generative engine optimization (GEO) and answer engine optimization (AEO).

Template: visible in AI search, opted out of training

# Search engines and AI citation bots: allowed
User-agent: Googlebot
User-agent: Bingbot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
Allow: /
Disallow: /cart/
Disallow: /checkout/
Disallow: /my-account/

# Training crawlers: blocked (policy choice)
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Bytespider
Disallow: /

# Everyone else
User-agent: *
Disallow: /cart/
Disallow: /checkout/
Disallow: /my-account/
Disallow: /search/

Sitemap: https://www.example.com/sitemap.xml

Step-by-step robots.txt setup

1. Inventory your URLs. List the sections that waste crawl budget (cart, internal search, filters, sessions) and those that earn revenue.
2. Check your current file. Open /robots.txt on every host and subdomain, and note the status code.
3. Write rules by intent. Use trailing slashes, wildcards and $ deliberately, then add Allow exceptions.
4. Set your AI policy. Decide search, training and user-fetch bots separately.
5. Add the Sitemap line with an absolute HTTPS URL.
6. Test before deploying. Use the Search Console robots.txt report and a crawler such as Screaming Frog in robots-testing mode.
7. Deploy and monitor. Watch server logs for the bots you care about and confirm they follow the new rules.

What an XML sitemap does

An XML sitemap is a machine-readable list of the canonical URLs you want search engines to discover, with optional last-modified dates. It helps most on large, new or poorly interlinked sites, which includes almost every e-commerce catalog. It is a hint, not a guarantee of indexing.

A sitemap index lets you split a big site into several child files:

<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <sitemap>
    <loc>https://www.example.com/sitemaps/products-1.xml</loc>
    <lastmod>2026-10-22T09:30:00+00:00</lastmod>
  </sitemap>
  <sitemap>
    <loc>https://www.example.com/sitemaps/categories.xml</loc>
    <lastmod>2026-10-20T14:05:00+00:00</lastmod>
  </sitemap>
</sitemapindex>

Sitemap rules to remember

Common sitemap errors and fixes

Step-by-step sitemap setup

1. Crawl your site and export every URL with status, canonical and indexability.
2. Keep only index-worthy URLs. Remove redirects, errors, duplicates, parameters and noindex pages.
3. Segment into files by template: products, categories, articles, images.
4. Automate generation in your CMS and set lastmod from the real update timestamp.
5. Publish at stable URLs and reference the index in robots.txt.
6. Submit and monitor in Search Console and Bing Webmaster Tools. Compare “submitted” with “indexed” per sitemap.
7. Re-audit monthly. Pair this with our technical SEO audit guide and the full technical SEO guide 2026.

What about llms.txt?

llms.txt is a proposed Markdown file that points AI agents to your most useful pages. It is cheap to add and sometimes read by agent tools, but no major search engine has confirmed it as a ranking or citation signal. Treat it as optional. Strong crawlability, clean sitemaps, structured data, clear answers and trusted mentions matter far more. See our AI SEO beginner’s guide for what actually earns citations.

Best tools for robots.txt, sitemaps and AI crawler control

Prices were checked in October 2026 and change often. Confirm on each vendor’s site before buying.

Frequently asked questions