Submitted URL marked ‘noindex’ in Google Search Console

In the newly released beta of Google Search Console, your site might have one or more results for this Error: Status > Index coverage > Submitted URL marked ‘noindex’

Status > Index coverage > Submitted URL marked ‘noindex’

This isn’t necessarily an error, although if a site is not being indexed properly, this is a great place to start.

A page that is set to ‘noindex’ (vs. the default ‘index’) requests to Google that the page not be indexed nor displayed in search engine results. Google may still crawl this page and follow the outbound links, and it may also display an abbreviated listing in the SERPS.

Here’s an example – our Client Portal is ‘noindex’ because it’s just for clients. There is no content that we want found in search results. You can see below that Google knows the site exists, and the home page is in Google’s index. Other than that, Google tells us that “no information is available for this page” which is fine by us.

noindex in search results

Why noindex?

In WordPress, sections of the site (be they a custom post type or part of the taxonomy) may be marked ‘noindex’ for various reasons, particularly if there is no additional information beyond a collection of links. Imagine a user landing on such a page and trying to make sense of it. If there is no additional value or context added, the user is less likely to engage, which is a lost opportunity and seen negatively by Google (via time on site, bounce rate, etc).

Check everything at site launch: An Asterisk Is Costing the U.S. Open Thousands of Visitors

When reviewing this “Submitted URL marked noindex” error in the new Google Search Console, first ensure nothing is marked ‘noindex’ that shouldn’t be. When a website is built, the entire site is generally set to “noindex” even if it is located on a password-protected server. So when that site launches, sometimes someone forgets to change that setting to “index” and the site can be hidden from Google indefinitely! See the article to the right for a great example of how to launch a website without remembering to check basic settings.

Search Console’s noindex Options
  • Test Robots.txt Blocking – as in the “U.S. Open” example above, the robots.txt file that sits in the root directory of your site can be the cause of a page or directory or entire site not being indexed. Checking noindex and robots.txt for potential directions to Google is essential.
  • Fetch as Google – see how Google sees your page on both desktop and mobile. You can click through to see the page code (what Google calls “Downloaded HTTP response”) and how it renders for Google vs. actual human visitors.
  • View as Search Result – because the page is set to “noindex”, this should display no results or possibly the obscured version we saw in the Client Portal example above.
  • Submit to Index – this would only be applicable if you have changed a page from “noindex” to “index” and subsequently want to make sure Google sees it. Generally, we’d want Google to organically crawl the newly indexed page via the website’s navigation and internal link structure, but using the “Submit to Index” option may be useful to force the issue.
When to Index noindex

Reviewing the noindex pages in Google Search Console shows us where there may be SEO opportunities ripe for the picking. This entails making certain pages ‘index’ and adding quality, useful content.

An Author page by default usually just contains links to that author’s posts. Looking at this from a user experience, how do we add value to an Author page that’s otherwise a bundle of links? We can add in a full bio for the author and a carousel of photos. This creates a much more well-rounded context for links to the author’s posts. We can also adjust the Page Title, the Meta Title and the Meta Description.

noindex author page in wordpress

For instance, those two elements (Page Title and Meta Title) might, by default, be named “Author Archives: Bob Smith” and could be better named something like “Bob Smith, Featured Author” or “Bob Smith, Chef and Food Critic”. Once the page is changed from “noindex” to “index,” it is now much better positioned to rank for relevant searches and to appeal to site visitors at the same time.

Categories and Tags

Categories and Tags are likely candidates for ‘noindex’ as well. This is because they simply aggregate a bunch of content without adding value. This SEO opportunity (which I refer to as Taxonomy SEO) is to add quality content that adds value to the collected posts on that topic.

For instance, let’s say a recipe website has 8 recipes that have been tagged with “carrot”. Visiting that page will show eight headlines, possibly with some additional metadata like an excerpt, author, date, and # of comments. It may have a Page Title and Meta Title of “Tag Archives: Carrot.” That’s about it. It’s not a compelling user experience.

How to add value? Start by adding 500 words or more to the top and/or bottom of the page, putting “carrot” in context – the culinary origins of carrots, different varieties of carrots, how the taste of a carrot is described, how carrots are used in both savory and sweet dishes, etc.

Handy for marketers: Social Network Image & Video Size Guide
Change the Meta Title, Meta Description and Page Title to more accurately describe the Carrot Recipes. Add some high-quality images large enough for social sharing.

Let’s kick it up a notch. You can then edit each article that’s been tagged “carrot” and write a custom excerpt of 50 words or so that is unique from the content on the page. This further architects a real, unique, interesting page of value to both Google and site visitors.

Bring all these elements together, and this page is “index” ready. Note the above changes may require a plugin or some developer time.

Review Your Sitemap

If your site has a sitemap, here’s a handy tip. Look at it! There may be any number of pages you don’t actually want on the sitemap (which basically tells a search engine spider what pages it can crawl). For instance:

  • test pages
  • old, outdated pages
  • administrative pages
  • confirmation pages

How do you know if you have a sitemap? First, try going to yoursite.com/sitemap.xml. If that doesn’t produce any results, check with your web developer, who can usually enable a sitemap with a quick plugin. Also of note, sitemaps can be submitted to Google Search Console for validation and crawling, although the strategy and efficacy of doing so is the topic of another post.

If your site has pages on the sitemap you don’t want visible, you’ll need to do two things:

  1. change the status of each page to ‘noindex’
  2. exclude the page from the sitemap

How you do both of these things are largely dependent on how your site is set up. In WordPress, the free Yoast SEO plugin is widely used for managing both of these things.

This noindex Summary

Google flagging ‘noindex’ pages as errors in Search Console can be quite revealing. Use this opportunity to make sure the pages that are effectively hidden should be. Also look for opportunities to build out great content for your visitors on pages that could be changed to ‘index’. And check out your sitemap – it can be pretty revealing in unexpected ways. Let us know in the comments below if you have any questions!

Comments 8

  1. Thanks for this! I was wondering if there was a way to tell Google to ignore the pages that are noindex on purpose? Should I try to remove them from the sitemap?

  2. Post
    Author

    Hi Eva – thanks for writing in! There are three primary ways to *discourage* Google from crawling and indexing specific parts of your site, be it the whole site, a folder or a page.

    1) noindex tells Google that you don’t want the page to appear in search results. That page may still be crawled, however, and pairing noindex with follow or nofollow suggests to Google whether or not it should continue crawling and whether or not your page’s authority should be passed on to the recipient of the link.

    2) removing a page from a Sitemap is an additional step you can take to minimize the chance Google will crawl and index the page, and can be paired with noindex. If you are on WordPress, try the Yoast SEO plugin, which has an XML Sitemaps tab. You can add the post ID for the page you don’t want in the sitemap, and then check it by going to yoursite.com/sitemap.xml.

    3) .htaccess is a file that sits in the root of your domain. You can use it to explicitly exclude Googlebot from accessing parts of your website (like /wp-admin/ or a client area for instance). You need to be really careful with what you put in your .htaccess file, since a wrong character can tell Google not to crawl your site at all!

    Hope this helps – let me know if you have additional questions.

  3. Hi;
    Many thanks for sharing this useful article;

    I read carefully, i have one problem in my tags, some of my tags in google search console are “no-index” , i checked everything but no result found.

    Just thinking about plugin conflict maybe caused this error, for example i used “404 redirect to home” plugin,
    I’m suspicious to four plugins installed:
    1- 404 To Homepage
    2- Social Media Auto Publish
    3-Wordpress Pinterest Automatic
    4- WP Super Cache
    Please share your experience about this issue if you have.

    Thanks again
    Sam

  4. Post
    Author

    Hi Saman – Thanks for writing in. A couple thoughts for you:

    1) setting all Tags to “noindex” can be a good strategy, if you are not adding good content to a collection of tagged posts. See my notes above on Taxonomy SEO. Either way, you want to know why some or all of your Tags are set to “noindex”. Since you are on WordPress, I would check to see if you have an SEO plugin like Yoast SEO or All-In-One SEO. If so, look for the Tag settings.

    In Yoast SEO, go to Search Appearance > Taxonomies. Find the Tags section. It should say “Show tag in search results?” If you expand the tool tip, you’ll see it says this: “Not showing Tags in the search results technically means those will have a noindex robots meta and will be excluded from XML sitemaps.” So you can decide to set noindex on your tags or not by toggling this setting. (Don’t forget to Save!)

    2) if you think there’s a plugin conflict, the common advice is to disable all your plugins and re-enable them one-by-one until you find the issue. You can certainly do that, and look in the source code of any of your Tag pages to see if they are set as “noindex” or “index” or no defining tag at all. If you feel that that the issue is with how Google is crawling or viewing your content, disable all four of them for a week or so, until you see how Google is viewing and indexing those pages.

    Hope this helps. Good luck!

  5. Google Search console is telling me that my sitemap.xml itself is marked ‘noindex’ – which is correct, right? I don’t want users looking at my ugly sitemap.xml file – I only want google (and friends) to look at it so that they can index the pages my site map is telling them are interesting. I have had this error on a couple of WP sites I am running. I presume I can just ignore this error, but don’t really want Google to keep bugging me about it if it isn’t actually an error. Is this something you have come across? Or that I can do about?

  6. Post
    Author

    Hi Charles – that forum post on Google you shared had a bunch of good information. I’d check your SEO or Sitemap plugin to see if either is producing a sitemap.html version in addition to sitemap.xml. Also, I’d recommend taking the time to claim your website in Google Search Console and submitting your sitemap, if you haven’t already. Try viewing the source of your sitemap.xml to see if there is a Robots meta tag or X-Robots-Tag HTTP header. Also take a look at your robots.txt file by going to yoursite.com/robots.txt and see if anything is being blocked that shouldn’t be. Hope this helps!

Leave a Reply

Your email address will not be published. Required fields are marked *