When your pages are not showing up at all, and how to find out

Before you assume you are ranking badly, check that you are ranking at all. Being absent and being buried look identical from where you are sitting, and they need completely different fixes.

There are two reasons you cannot find your page in a search result, and from where you are sitting they look exactly the same.

Either the page is in the index and sitting a long way down, or it is not in the index at all. The first is a competitive problem that takes months of patient work. The second is usually a switch in the wrong position, and fixing it can change things within a fortnight.

It is worth ten minutes to find out which you have, because a great deal of money gets spent on the first problem by businesses who actually have the second.

Indexed, and ranking, are different things

Crawling is a search engine visiting your page. Indexing is it deciding to keep the page. Ranking is where the page appears once somebody searches. A page has to pass through all three, in order, and most conversations about search skip straight to the third.

The quick test is the one in the first part of this series: search for a distinctive sentence from your page, in quotation marks. If the page comes back, it is indexed and you have a ranking problem. If it does not, keep reading.

You may also have seen the advice to search for site: followed by your domain. It is a useful rough count and it is not authoritative, so do not read too much into the exact number it gives you.

Search Console is where the real answer is

Google’s Search Console is free, it is your own data about your own site, and it takes a few minutes to verify. Nothing else gives you the same view.

Two parts of it matter here.

The pages report splits every URL Google knows about into indexed and not indexed, and for the second group it tells you why. That reason is the whole game, and the common ones each mean something specific.

URL inspection does the same for a single address. Paste in a page you are worried about and it will tell you whether it is indexed, when it was last crawled, and what is stopping it if it is not.

What the common reasons actually mean

Excluded by “noindex” tag. You have told search engines not to keep this page. Occasionally deliberate, often left over from before launch.

Blocked by robots.txt. You have told them not to visit. Note the difference from the one above: this stops the crawl, not the indexing, and a blocked page can still appear as a bare result because the instruction not to index it was never seen.

Crawled, currently not indexed. Google looked and decided not to keep it. This is a judgement, not a fault, and it usually means the page is thin, close to identical to another page, or has no obvious reason to exist. More pages is not the fix.

Discovered, currently not indexed. Google knows the address and has not got round to it. Common on large sites and on new ones, and often a sign that nothing links to the page.

Alternate page with proper canonical tag. The page is pointing at another page as the version to keep. Correct behaviour if you meant it, and a common accident if you did not.

Duplicate, Google chose a different canonical. You did not specify, and Google picked. Worth checking it picked the one you would have.

Page with redirect and not found (404) mean what they say, and are only a problem if the address was supposed to work.

The classic own goal is blocking a page in robots.txt to keep it out of search. Because the crawler never visits, it never sees the noindex instruction you also added, so the page can linger in results with no description under it. If you want a page gone from search, let it be crawled and use noindex. If you want it never visited, use robots.txt. Doing both to the same page is the one combination that does not work.

The reasons that are not about switches

Nothing links to it. A page that no other page on your site points at is hard to find and easy to ignore. Internal links are the cheapest indexing fix there is.

It is behind something. A login, a form, a filter that only exists once JavaScript has run. If a visitor has to do something to reach it, a crawler probably cannot.

It is one of forty near-identical pages. A page per town, or per tiny product variation, with the same words and one word swapped. Google is quite good at spotting these and will keep one of them at most.

Your site is very new. Indexing takes time, and a site with no history and few links can take weeks. Patience is a legitimate answer, though it is worth being sure that is what you are looking at.

Sitemaps help, within limits

An XML sitemap is a list of the pages you would like considered. Submitting one in Search Console is worth doing, and it particularly helps a large site or a new one.

What it does not do is force anything. A page in your sitemap can sit unindexed indefinitely, and a page missing from your sitemap can be indexed perfectly well. It is a suggestion, not an instruction, and it is not a substitute for the page being linked to from somewhere.

Two things worth checking on yours. That it only lists pages you actually want indexed, since sitemaps generated automatically often include tags, archives and other clutter. And that the addresses in it match the ones your site actually serves, including whether they have a trailing slash and whether they are the www version.

A sensible order to work through

Confirm the problem first with the quoted-sentence search, so you know whether you are diagnosing absence or position.

Then open the pages report and read the reasons rather than the total. One reason usually accounts for most of the excluded pages, and it is normally something with a single cause.

Fix the switches first, because they are quick and unambiguous. Then look at the pages Google crawled and declined, and be honest about whether each one deserves to exist. Merging six thin pages into one good one does more than any amount of resubmitting.

Then wait. Recrawling takes days to weeks, and requesting indexing for a handful of important pages is reasonable while doing it forty times is not.

Where to start

If you have never verified Search Console, do that today. It is free, it is the only place this information exists, and every conversation about search you have from here on will be better for it.

Then read the pages report once, slowly, and write down the top three reasons by volume. That list is usually shorter and more fixable than people expect.

If you would rather somebody went through it with you and told you which of it matters, get in touch. Making sure a site can actually be read and kept is part of how we approach search, and it is always the first thing we look at.