Restricting search engine spiders
Restricting search engine spiders while making quality choices can improve SEO.
- Meta tag placed in the HTML header. It tells the spiders which URL is the canonical version of the page being displayed.
<link rel="canonical" href="http://example.com/good_page.htm"/>
- A 301 redirect is a better solution.
- File in the root directory of the website that indicates what should be indexed and what not.
- Better to use the meta tag noindex
Meta Robots Tag
- The meta robots tag creates page-level instructions for search engine bots. This is the preferable option.
- Meta robots tag syntax
page revision: 2, last edited: 19 Mar 2011 11:05