top of page
  • Writer's pictureBrent Payne

Disallowed URL in XML Sitemaps

When a URL that is blocked by the robots.txt file appears in an XML Sitemap, it poses a conflict for search engines.


Why is this important?

Your XML Sitemap should only include URLs that you want search engines to crawl and index. If you disallow a URL, you're telling search engines not to access it, and thus not to index its contents.


Consequently, placing disallowed URLs in your sitemap sends mixed signals to search engines, which could result in these pages appearing in search results, incorrectly:


What does the Optimization check?

This Optimization will be activated if any internal URL that is disallowed by the robots.txt file is discovered in the XML Sitemap.


Examples that trigger this Optimization:

Suppose you have the URL: https://example.com/pages/page-a in an XML Sitemap submitted to search engines.


This URL would cause the Optimization to trigger if it was found to match a disallow directive in the robots.txt file, such as:

User-agent: *Disallow: /pages/page-a


How do you resolve this issue?

The inclusion of disallowed URLs in your sitemap is not correct practice and can lead to confusion in indexation. To correct this, do either:

  1. Remove the disallowed URL from all XML Sitemaps, then resubmit the updated sitemap to search engines via tools like Google Search Console.

  2. If the URL should no longer be disallowed, amend or eliminate the relevant disallow directive in the robots.txt file.

Further Reading

5 views

Recent Posts

See All

ClubReq Gets A Link (Because They Asked)

I am a mentor for Techstars and have been for over 10 years. In those ten years I have mentioned to startups to ask ANYONE and everyone that they meet to link to their site. Yet, in all those times on

Comments


bottom of page