Google Supplemental Pages

You'll find the latest information about Google's Supplemental Index and Supplemental Pages at the SEO Theory blog (written by Michael Martinez). Check out Google Broken: Supplemental Pages Not Being Parsed And Indexed, Google's Web Apartheid: Gone Supplemental And Getting Nowhere, and Why you now have Supplemental Results pages on Google.

Why Do Some Pages Say "Supplemental Result" In Google's SERPs?

Google maintains many Web page databases. Their specialized searches (e.g. News Search, Blog Search, etc.) operate on separate databases. But the Web search service itself is operated against two primary databases: the Main Index and the Supplemental Index.

There is a great deal of speculation and misinformation in the SEO community about the Google Supplemental Pages Index. Outside of the official Google answer to the question, The only four people who have spoken with authority on the topic are Google employees: the ubiquitous GoogleGuy, Matt Cutts, Vanessa Fox, and Adam Lasnik.

What Is The Purpose of the Google Supplemental Pages Index?

When the Google Supplemental Pages Index first appeared in the summer of 2003, GoogleGuy indicated that its purpose was to expand results for obscure queries. Mark Carey has collected many of GoogleGuy's comments about the Supplemental Index.

When discussing the early 2006 update dubbed Bigdaddy, Matt Cutts mentioned the Google Supplemental Pages Index. While he did not explain it, Matt did say "personally I’d think of [the Supplemental Index] as a fallback way that we can return results for specific queries where we might not have as many results in the main index." While that agrees with what GoogleGuy said in 2003, it doesn't explain what the Supplemental Index really is. However, Google's Web site says "We're able to place fewer restraints on sites that we crawl for this supplemental index than we do on sites that are crawled for our main index." The same document also says "be assured that the index in which a site is included doesn't affect its PageRank."

Inundated by requests for information, Adam Lasnik wrote in Google Groups that the "cure" for being in Google's Supplemental Results pages is to "get more quality backlinks". Of course, he doesn't explain what a "quality backlink" is.

If all it takes to move from Supplemental to Main index is "quality backlinks", then why does Google need a Supplemental Pages Index at all? In February 2005, Mike Grehan interviewed Apostolos Gerasoulis from Ask, who claimed that Google has not fully implemented PageRank. Maybe that's the clue. Maybe the Google Supplemental Pages Index consists of pages that, for whatever reason, cannot confer PageRank and anchor text.

How Does Google Decide Which Pages Should Be Supplemental?

Google claims the process is automatic. Some of the best-known voices in the SEO community suggest the following reasons for why pages may go into the Supplemental Index:

1. Duplicate content. If Google already sees the content on one page, why should it include another page with the same content in the Main Index. While this seems logical, there are many Wikipedia scraper sites whose pages are in the Main Index. Simply having duplicate content does not appear to be sufficient reason for a page to go into the Supplemental Index.
2. Canonical issues. If you link to your page with the "www" prefix, without the "www" prefix, with the page name (ala "index.html") and without the page name (ala "http://www.example.com/"), you have created a canonical issue. To a search engine, those look like four different pages, when in fact they are one. However, many pages end up in the Main Index with multiple URLs.
3. Insufficient links to the domain. I have found many deep, secondary pages listed in the Main Index while the majority of their pages (including the root URL) are in the Supplemental Index.
4. Distance from root URL. The idea is that the more directories deep on your site a page is, the more likely it is to show in the Supplemental Index. Many SEOs are skeptical because we have all seen deep content rank highly in search results. A rule-of-thumb that has been gaining in popularity is the more clicks it takes to reach a page from the root URL, the more likely it is to show as Supplemental (and the less likely it is to be indexed at all). However, deep content can have more external linkage than internal linkage. On Xenite.Org, the pages that show as Supplemental tend to have no more than 1 link pointing to them regardless of where they fall in our directory hierarchy. They are usually no more than 3 or 4 links from the root URL.

Matt Cutts did say in October 2006 that PageRank is the primary factor determining whether a url is in the main web index vs. the supplemental results, so I’d concentrate on good backlinks more than worrying about varying page layouts, etc.. Some people seem already to have forgotten about the "primary" in his comment, but nonetheless when Matt Cutts talks about PageRank he is almost invariably talking about "internal" PageRank, not the silly number you see in the Google Toolbar.

The best guess I can make is that a page goes Supplemental either because it's a new page that has only been crawled by the Supplemental Index's robot (Matt Cutts confirmed the indexes use their own crawlers) or because it has too few "trusted" links pointing to it. I think that pages are only placed in the Main Index if they are found through a minimum number of links already in the Main Index.

Does Being Shown As Supplemental Hurt A Page's Ranking?

The implication from Google and its representatives is that Supplemental pages are more likely to be shown for obscure queries that have few if any results in the main index. Prior to September 2006, it seemed highly unlikely that being listed only in the Supplemental Index hurt a page's chance to rank well in search results. A well-populated query will return 1,000 results from a much larger pool of results -- even so, you could still see Supplemental Index pages in many results sets.

However, in September 2006 the Search Engine Roundtable reported that Google's cache was not highlighting query words for Supplemental Index pages. In February 2007, Michael Martinez confirmed on the SEO Theory blog that Supplemental Results pages were not appearing for unique on-page text queries. It therefore appears that since Google's Summer 2006 update Google has not been parsing and indexing Supplemental Results pages. And that means a page showing as Supplemental will not rank for any meaningful queries (they do appear in site search results).

Prior to the Summer 2006 Google Update, the query resolution process sought out the most appropriate pages from both the Main Index and the Supplemental Index. Query results are ordered by Relevance Scores, and Relevance is determined by a large number of factors (none of them having anything to do with number of links). Supplemental Results pages, because they were parsed and indexed, had a chance to rank. But until Google reactivates that feature for the Supplemental Results Index, it should be assumed that Supplemental Results pages probably will not appear in most queries.

How Do You Move Pages Into The Main Index?

Try getting links from pages that are already in the Main Index. You should not need many. We have been able to move pages into the Main Index with fewer than 10 links. Try to get links from pages that don't sell links, not because paid links are bad but because a page that doesn't sell links is more likely to pass value than a page that does sell links. You have no way of knowing which pages pass value.

If you have a large content site, you need to have multiple secondary indexes -- one for each sub-section of the site, at least -- as well as extensive cross-linking between your pages. You cannot have 1 page link to 1,000 others. On many of Xenite's pages we do actually provide extensive link sections to help people find related content on Xenite.Org and throughout our network. Nonetheless, many of our pages just rely on links back to the root URL and our Site Map pages. The tradeoff is that with less interlinking, some sections are more likely to have content that goes Supplemental. We still earn a lot of organic traffic from search engines, so we're not really concerned.

How Long Does It Take To Change Indexes?

We have seen results in 3 to 10 days.

Why Do Pages Go Back Into Supplemental Results?

Google is constantly crawling the Web. It seems very likely that pages are automatically delisted from the Main Index if, after some period of time, they are not reached again through trusted links. If we assume for the sake of discussion that is the case, the more links you have pointing to a page, the more likely it is to stay in the Main Index. However, one good link is worth 1,000 cheap links. Many of Xenite's pages stay in the Main Index even through strenuous updates, and most of our pages have fewer than 5 inbound links, almost all of them internal.