What is Robots.txt and How to create?

Robots.txt is a standard content document for speaking with web Crawler/Spider/bots and train them to which region of site or site page is crept or not. Robots.txt is a Publicly accessible document and anybody can see effectively with part or url's of site crawler output or not. As a matter of course Search Engine Crawler creep all that they conceivable.


How to create robots.txt ?

There are numerous online instruments for making robots.txt record furthermore you will make this document manually.Before making robots.txt record you will see a few guidelines and regulation. I have disclosed here how to make a robots.txt record

In the robots.txt document USERAGENT line recognizes the web crawler and DISALLOW: line characterizes which some portion of the webpage is refused 

[1] Here is the Basic robots.txt document 

User-agent: *
Disallow : /

In the Above assertion "*" demonstrate the all crawler/arachnid/bots and "/" characterize every one of the pages are prohibited. We can say it at the end of the day, that the entire destinations are refused for all crawlers.

[2] on the off chance that you need to prohibit any particular web crawler or insect not slither your website, then the robots.txt record will be 

User-agent: Yahoobot
Disallow: /

In above illustration I utilized Yahoobot and you will utilize that crawler which you not have any desire to creep you site.

[3] If you need to diallow all crawler for particular envelope or website pages then 

Client specialists: * 

Disallow: cgi-bin
Disallow: abc.html

[4] If you need to forbid all crawler for entire site with the exception of any particular crawler for permitted slithering then 

User-agent: *
      Disallow: /
      User-agent: Googlebot
      Disallow: /cgi-bin
      Disallow:/ abc.php

[5] in next case you can utilize refuse to transform into permit standard by not entering any quality or/after semicolon (:) . 

     User-agent: *
     Disallow :/
     User-agent: Googlebot
     Disallow :

Labels: