Seach Engine Facts

How are search queries resolved?

When you type a query into a search engine like Ask, Google, MSN, or Yahoo!, the engine selects a number of documents (Web pages) from its index that use or are associated with the words in your query. The documents are assigned relevance scores, usually on the basis of how much the words are used, how close they are to each other, whether they occur in the same order, if they are used in title elements, Hx header elements, large font sizes, or otherwise given special emphasis such as being used in link anchor text (inbound or outbound), bolded, italicized, colored, etc.

These scores may be cumulative by occurence. That is, the number of times a word appears in the document, is bolded, appears in link anchor text, etc. may be significant. The search engines also look at how important the pages are. Importance is usually determined on the basis of linking relationships. Google and MSN both claim to use PageRank. Yahoo! appears to also use PageRank or some other link poplarity model. Ask uses a more refined system called ExpertRank.

The combined Relevance and Importance scores are used to sort the selected results pages.

How do search engines evaluate the "robots" meta tag "noindex" attribute?

Matt Cutts published an interesting case study of the NoIndex value for the robots meta tag. He found that neither Ask nor Google will include a page in its index if they encounter "noindex" in the robots meta tag. MSN will include the URL of the page in its index but provides no other information. Yahoo! apparently ignores the value altogether.

Which search engines support rel="nofollow"?

There is not much agreement on which services have actually implemented support for the rel=nofollow link attribute. However, given Matt Cutts' frequent suggestions to Webmasters that they use it for paid links, it seems clear Google is supporting the atttribute. Ask declared they had no need to support it. MSN and Yahoo! originally agreed to support the attribute but neither search engine mentions it in their current guidelines as Google does.

How long does it take to get indexed?

Since late 2004, I have seen Ask, Google, MSN, and Yahoo! add pages on my sites to their indexes in about 1-3 weeks. Yahoo! is the slowest. MSN indexed the main page for this site within a day. Yahoo! indexed one reference to seo.xenite.org within about two days. Google and Ask have yet to show anything in their index as I write this page. All four services have at least fetched the main page.

The time it takes a new Web site to be indexed can be divided into two segments: first, how long does it take to be crawled? Secondly, what is the lag time between crawling and indexing? In March 2006, I announced a Sandbox test for another domain. I had actually registered the domain in August 2005 and had to put up a robots.txt file to send the crawlers away. In that test, Google crawled the site as soon as I opened it up, but I didn't see the site listed for at least a week. MSN quickly indexed the site. Yahoo! had picked up the site within 2 weeks.

The amount of time it takes for a new domain to be crawled is probably not significant. You should have content ready to go up on your new domain as soon as you purchase it. The crawlers will come back often. If you create a site as a sub-domain or just add new pages to an existing domain, you'll have to wait for the spiders to find the site through links (unless you can get them to respond to a URL submission, but Ask won't even let you do that). I was able to springboard seo.xenite.org by placing links on Xenite and a few other trusted sites.

How long does it take to get a high ranking?

That's the million-dollar question, isn't it? Many people report that it takes 1-2 years. I have seen many sites achieve useful high rankings in far less time. But there is no set timeframe, no schedule. Your site will rank for a variety of reasons. It won't rank for a variety of reasons. Search engine optimization requires testing and consideration of many factors.