Correct Use of Content Deduplication and Canonical Tags
Chapter 4: Content Marketing · Question 63
You may not have "plagiarized" anyone, but your site likely already has a large amount of duplicate content—the same article accessible via different URLs, and search engines don't know which one to index, ultimately diluting the rankings of all versions.Canonical tagsare the core tool for solving this problem.
What is duplicate content?
Duplicate content isn't necessarily "plagiarism"; more often it's technical reasons that cause the same content to have multiple URLs:
| Causes | Example |
|---|---|
| HTTP/HTTPS coexistence | http://example.com/page and https://example.com/page |
| www vs. non-www | www.example.com/page and example.com/page |
| URL parameters | /product?color=red and /product?color=red&ref=banner |
| Paginated content | /article and /article?page=1 |
| Separate mobile URLs | m.example.com/page and www.example.com/page |
| Print pages/AMP versions | /article/print and /article |
How Baidu handles duplicate content
Baidu selects one of multiple duplicate URLs as the "representative" for indexing, and other versions are filtered or demoted. But Baidu's judgment isn't always accurate—it may pick the version you don't want. Actively using canonical tags tells Baidu: “please use this URL as the authoritative one.”
Note: Baidu's official documentation explicitly states support for canonical tags, but processing may not be as timely as Google's.
Correct syntax for canonical tags
In the HTML <head> section, add:
<link rel="canonical" href="https://www.example.com/preferred-url" />
Usage rules
- Must be placed in
<head>Medium: placing it in<body>is invalid - Use absolute URLs: write the full
https://www.example.com/path, don't use relative paths - Only one per page: Multiple canonical tags cause all of them to be ignored
- Point to an accessible page: Do not point to a 404 or robots.txt blocked page
The importance of self-referencing Canonical
Even if a page has no duplicate versions, it is recommended to add a Canonical tag pointing to itself:
<!-- 在 https://www.example.com/blog/seo-tips 页面上 -->
<link rel="canonical" href="https://www.example.com/blog/seo-tips" />
Why? To prevent malicious parameter propagation in the future (e.g., ?utm_source=xxx), and also to prevent search engines from becoming confused by crawling parameter changes.
Common mistakes
✗ Mistake 1: Canonical points to the wrong page
Category page A's canonical points to category page B, causing A to disappear entirely from search results.
✗ Mistake 2: Misuse on paginated content
Canonicalizing pages 2, 3, and 4 all to page 1 — this prevents subsequent pages' content from being indexed. Pagination should use rel="next/prev"(supported by Baidu) or keep each page independently indexed.
✗ Mistake 3: Canonical conflicts with robots
The page is blocked from crawling by robots.txt but still has a canonical set — crawlers cannot see this tag at all.
✗ Mistake 4: Cross-domain Canonical misuse
Canonicalizing your own site's page to someone else's site is equivalent to "giving up" all ranking weight for that page.
301 redirect vs Canonical: How to choose?
| Scenario | Recommended solution |
|---|---|
| Old URL permanently retired | 301 redirect |
| Both URLs need to be accessible to users | Canonical tags |
| HTTP migration to HTTPS | 301 redirect |
| Same page with and without parameters | Canonical tags |
| Website redesign with domain change | 301 redirect |
Simple principle: If users do not need to access the old URL, use 301; if both URLs need to exist (such as filter parameters), use Canonical.
How to audit a website for duplicate content
- Baidu SearchFrame detection: Search
site:yoursite.com + 文章标题关键句, to see if there are multiple results - Use crawler tools: Tools like Screaming Frog and 5118 can batch-detect duplicate Titles and duplicate content pages
- Baidu Search Resource Platform: Check the difference between "index volume" and the actual number of pages; a large gap indicates many pages are not indexed (possibly due to duplication)
- Server log analysis: Observe whether Baidu Spider is wasting crawl quota on duplicate URLs
Practical recommendations
- Deploy self-referencing Canonical site-wide: This is the lowest-cost, most reliably beneficial basic SEO operation; have developers add it uniformly at the template level
- Prioritize parameter-based duplication: In the "parameter configuration" feature of Baidu Search Resource Platform, mark meaningless parameters (such as session ID and tracking codes) to tell Baidu to ignore these parameters
- Conduct a duplicate content audit once a month: Use crawler tools to scan the entire site, focusing on page pairs with similarity above 80%, and decide whether to merge, add Canonical, or rewrite for differentiation
Use AI tools to solve: duplicate content audits and Canonical configuration
Many of the steps above can now save most of the manual work with AI tools, especially the two types of work: "analysis" and "generation." Below are ready-made tool combinations and prompts you can copy directly, organized by scenario.
Scenario 1: Batch detection of duplicate content
The most labor-saving combination:Crawler tools (Screaming Frog / 5118) export data + general AI (ChatGPT / Claude / Kimi / Doubao) analysis。
Method: Use a crawler tool to crawl the entire site and export a CSV containing URL / Title / 字数 / 描述 (Screaming Frog's View > All URLs or Export > CSV), paste the CSV content to the AI, then copy this prompt:
I uploaded a website crawl report with fields including URL, title, and word count. Please help me find page pairs with suspected duplicate content. Criteria: title similarity above 90%, or similar word count with similar URL structure. For each group, give a recommendation: merge into one page / add canonical pointing / rewrite for differentiation, and explain the reason.
The AI will list a checklist with recommendations according to your criteria. You only need to manually review the page pairs it filters out, instead of scanning hundreds of rows of data yourself.
Scenario 2: Generate Canonical code
If you're not sure how to write canonical in your framework, just ask the AI:
My website is [Hugo / WordPress / custom PHP], and the template file is at [path]. Please generate code for me that "automatically outputs a self-referencing canonical tag on every page site-wide." Requirements: use the absolute URL of the current page, place it in
, and output only on non-404 pages.
General AI is very familiar with canonical implementations for mainstream website-building frameworks, and the generated code is basically ready to use. Just remember to have a developer colleague take a look before going live.
Scenario 3: Sort out URL parameters and decide which ones to ignore
Feed the AI a list of URLs exported from server logs or an analytics dashboard:
This is a list of site URLs, including many URLs with parameters (such as ?color=red, ?ref=banner, ?session_id=xxx). Please help me classify them: which parameters affect page content (and must retain a separate index), which are tracking/session parameters (that can be ignored via canonical or parameter configuration), and provide a list of regular expressions.
Scenario 4: Generating a 301 mapping table during a site redesign / migration
My old site has 2,000 URLs about to be migrated to a new site. The old site URL list is as follows (first 50 pasted), and the new site URL rule is: [rule]. Based on the principle of "closest semantics", please generate a corresponding new URL for each old URL, output it in CSV format as oldURL,newURL, and provide an nginx configuration example in the form of 301 rewrite rules.
Scenario 5: Differentiated rewriting of duplicate content
When you encounter two pages with high similarity but both worth keeping, have the AI rewrite one of them from a different angle:
The following two passages are highly similar. Please rewrite the second passage from the angle of "practical steps", keep the first passage from the angle of "concept explanation", retain the core information but make the sentence patterns, structure, and examples all different. After rewriting, please self-check: do the two passages still have more than 80% consecutive duplicate sentences? (Paste the two passages)
Note: the boundaries of AI
- AI can analyze and generate, but it cannot execute for you: for canonical tags to actually go live, you still have to change the server/template; 301s require touching nginx, which must be done by someone with server permissions
- AI's judgments need review: the "duplicate page pairs" it lists may contain false positives (for example, paginated pages being misjudged), so manual review is essential
- Don't let AI directly invent 301 targets: the mapping relationship depends on your understanding of the business; AI can only guess based on "similar semantics", and the mapping of important pages must be confirmed by you
- Don't paste entire site HTML into free AI: there is a limit on the content pasted at one time, so feed it page by page; for sensitive data, use domestic compliant tools
