top of page
  • Writer's pictureBrent Payne

Disallowed URL in XML Sitemaps

When a URL that is blocked by the robots.txt file appears in an XML Sitemap, it poses a conflict for search engines.

Why is this important?

Your XML Sitemap should only include URLs that you want search engines to crawl and index. If you disallow a URL, you're telling search engines not to access it, and thus not to index its contents.

Consequently, placing disallowed URLs in your sitemap sends mixed signals to search engines, which could result in these pages appearing in search results, incorrectly:

What does the Optimization check?

This Optimization will be activated if any internal URL that is disallowed by the robots.txt file is discovered in the XML Sitemap.

Examples that trigger this Optimization:

Suppose you have the URL: in an XML Sitemap submitted to search engines.

This URL would cause the Optimization to trigger if it was found to match a disallow directive in the robots.txt file, such as:

User-agent: *Disallow: /pages/page-a

How do you resolve this issue?

The inclusion of disallowed URLs in your sitemap is not correct practice and can lead to confusion in indexation. To correct this, do either:

  1. Remove the disallowed URL from all XML Sitemaps, then resubmit the updated sitemap to search engines via tools like Google Search Console.

  2. If the URL should no longer be disallowed, amend or eliminate the relevant disallow directive in the robots.txt file.

Further Reading

1 view0 comments

Recent Posts

See All

CSS file size too large discovered in SEO audit

In our SEO audits at Loud Interactive, we often come across a common yet significant issue that can heavily impact your website's loading speed and overall performance: CSS file size too large. Let's

CSS file size too large discovered in SEO audit

When conducting an SEO audit on your site, one issue that might come up is that your CSS file size is too large. But why does this matter, and what can you do about it? Let's dive in. What's the Issue

H1 tag missing or empty" Discovered in an SEO Audit

At Loud Interactive, we've encountered a common issue during our SEO audits that can significantly impact a website's performance on search engines: the "H1 tag missing or empty" warning. This problem

bottom of page