When a canonical tag points to a URL that is disallowed by the site's robots.txt, this creates a conflict for search engines trying to determine the authoritative page.
Why is this important?
Canonical tags are a directive to search engines to indicate the preferred URL for content. If the canonical URL is disallowed, search engines receive mixed signals, often leading them to disregard the canonical direction and make their own judgment on which URL should be canonical. This could result in search engines indexing the wrong URL, potentially causing issues with duplicate content.
What does the Optimization check?
Loud Interactive's Optimization will activate if any page on your site includes a canonical link that points to a disallowed URL according to your site's robots.txt file.
Examples that trigger this Optimization:
Take for example the URL: https://example.com/page-a
If this URL had a canonical link pointing to another page within the site,
<link rel="canonical" href="https://example.com/page-b" />
where the canonical URL https://example.com/page-b is disallowed in robots.txt:
User-agent: *Disallow: /page-b...
How do you resolve this issue?
To fix such conflicts, you need to verify whether Page B is the correct canonical page for Page A. If Page B is correct, then you will need to amend the robots.txt file to remove the disallow directive for this URL. Should Page B not be the appropriate canonical target, the canonical tag on Page A should be adjusted to refer correctly either to itself or to an alternative appropriate URL. Decisions on resolving such conflicts should be made on an individual basis.