Restricting search engine spiders

Introduction

Restricting search engine spiders while making quality choices can improve SEO.

Canonical Tag

  • Meta tag placed in the HTML header. It tells the spiders which URL is the canonical version of the page being displayed.

Example:

<link rel="canonical" href="http://example.com/good_page.htm"/>

Robots.txt

  • File in the root directory of the website that indicates what should be indexed and what not.
  • Better to use the meta tag noindex

Meta Robots Tag

  • The meta robots tag creates page-level instructions for search engine bots. This is the preferable option.
  • Meta robots tag syntax