Proper Configuration of robots.txt and sitemap

Chapter 2: Search Fundamentals · Question 30

robots.txt and sitemap.xml are the two foundational files through which a website "communicates" with search engines. The former tells crawlers "what not to crawl," while the latter tells crawlers "please come crawl these pages." When configured properly, they can greatly improve indexing efficiency; when configured incorrectly, they can cause an entire site to disappear from search results.

Basic robots.txt Syntax

robots.txt must be placed in the website root directory (e.g., https://example.com/robots.txt), basic syntax:

User-agent: Baiduspider
Disallow: /admin/
Disallow: /search?
Allow: /

User-agent: Googlebot
Disallow: /admin/
Allow: /

Sitemap: https://example.com/sitemap.xml

Key rules:

• User-agent: specifies which crawler this applies to (*means all)
• Disallow: disallowed crawling paths
• Allow: allowed crawling (takes priority over Disallow)
• Sitemap: declares the sitemap address

Which paths should be blocked?

Never block:

• CSS and JS files (Baidu needs to render pages)
• Image directories (unless you don't want images indexed)
• Mobilepages

sitemap.xml Format and Specifications

Standard sitemap format:

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://example.com/page1</loc>
    <lastmod>2025-06-15</lastmod>
    <changefreq>weekly</changefreq>
    <priority>0.8</priority>
  </url>
</urlset>

Best practices:

• Each sitemap file must not exceed 50,000 URLs and must not exceed 50MB in file size
• When exceeding the limits, split using a sitemap index file:

<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <sitemap>
    <loc>https://example.com/sitemap-posts.xml</loc>
    <lastmod>2025-06-20</lastmod>
  </sitemap>
  <sitemap>
    <loc>https://example.com/sitemap-products.xml</loc>
    <lastmod>2025-06-18</lastmod>
  </sitemap>
</sitemapindex>

Submission channels

• Baidu SearchResource platform: Search service → Standard indexing → sitemap submission (supports txt and xml formats)
• Google Search Console: Indexing → Sitemaps → Add new sitemap
• Bing Webmaster Tools: Sitemaps → Submit sitemap

Baidu also supports real-time submission of new page URLs via the "active push API", which is especially useful for time-sensitive content (such as news).

Recommended sitemap update frequency

Common mistakes

1. robots.txt has Disallow: / written but forgotten to change it back: The entire site is blocked — this is the most catastrophic error
2. sitemap contains URLs blocked by robots: Logically contradictory, search engines will be confused
3. sitemap contains 404 pages: Wastes crawling quota
4. Fake lastmod dates: Writing the current date every time causes search engines to ignore this field
5. sitemap location not declared in robots.txt: Not required, but recommended
6. sitemap encoding error: Must use UTF-8 encoding; special characters in URLs need to be escaped

Practical recommendations

• Must-check robots.txt before launch: Many websites use it during development toDisallow: /block the entire site, then forget to change it after launch, resulting in months without indexing
• Only put canonical URLs you want indexed in the sitemap: Don't include duplicate parameterized pages, noindex pages, or redirect source URLs
• Check sitemap health once a month: Scan with tools like Screaming Frog to ensure the sitemap contains no 4xx/5xx error pages

Solve with AI tools: robots.txt and sitemap configuration generation

Scenario 1: Generate robots.txt based on site type

My website is [corporate site/e-commerce/content blog], built on [WordPress/custom], directories that don't need blocking are [CSS/JS/images], directories that need blocking are [admin/search pages/parameter pages]. Please generate a complete robots.txt for me: write separate sections for Baiduspider and Googlebot, block admin and internal search, Allow CSS/JS/images, and declare the sitemap address at the end. Also comment on the purpose of each rule so I can understand it.

Scenario 2: Generate sitemap.xml from an existing URL list

This is the full URL list of my website (paste sitemap or URL list). Please help me: 1) Filter out URLs with parameters or containing admin/search that shouldn't be indexed; 2) Generate XML per the sitemap spec, each entry containing loc, lastmod (using the given date), changefreq, priority; 3) If over 50,000 entries, split into sitemap index format.

Scenario 3: Check for "landmines" in the sitemap

This is a sitemap.xml (paste content). Please help me check: 1) Whether all URLs match the allow rules in robots.txt (I'm also pasting robots.txt); 2) Whether any noindex pages or 301 redirect sources are included; 3) Whether lastmod is reasonable (are they all today's date?); 4) Whether XML encoding and escaping are correct.

Scenario 4: Generate regex to block parameterized duplicate URLs

My website URLs carry these parameters: ?sort=, ?filter=, ?page=, ?utm_source=, ?session_id=. Please generate regex rules for robots.txt to block these parameter pages, and explain the matching scope of each rule and possible false positives.

Note: the boundaries of AI

• robots.txt is a "veto" configuration: After AI generation, be sure to manually review line by line, especially ensure there is no Disallow: / site-wide blocking; before launch, visit robots.txt directly in the browser to confirm
• sitemap content must be real: AI can only generate based on the URLs you paste; it won't know which pages are noindex or deleted—you must filter them yourself first
• Don't let AI fabricate lastmod: Fake lastmod will be ignored by search engines and reduce trust; dates must come from your actual file modification times