Restricting search engine spiders


Restricting search engine spiders while making quality choices can improve SEO.

Canonical Tag

  • Meta tag placed in the HTML header. It tells the spiders which URL is the canonical version of the page being displayed.


<link rel="canonical" href=""/>


  • File in the root directory of the website that indicates what should be indexed and what not.
  • Better to use the meta tag noindex

Meta Robots Tag

  • The meta robots tag creates page-level instructions for search engine bots. This is the preferable option.
  • Meta robots tag syntax