What is Robot.txt in SEO?
chrispitt -2 Light Poster
Recommended Answers
Jump to PostIt is a file that tells a crawler how it should behave on your site.
Search engines send out web crawlers, also known as bots to websites which follow links and collect bits of information such as the number of images and meta tags.A robot.txt is a text file …
Jump to PostThe file goes in the root directory of your website, yes.
All 7 Replies
Octet 45 Newbie Poster Featured Poster
It is a file that tells a crawler how it should behave on your site.
Search engines send out web crawlers, also known as bots to websites which follow links and collect bits of information such as the number of images and meta tags.
A robot.txt is a text file on the web server which tells these crawlers what they can and cannot do (although they don't have to be followed). An example would be to tell them not to index certain pages or even entire directories.
In SEO, they can be useful for stopping them attempting to visit pages which require logging in, or to stop them visiting pages which are X months old. This way, the search engines only view the latest content that you have selected.
ravinder yadav commented: thanks for answer but do reply my quary +0
allenhill99 0 Newbie Poster
robots.txt is a simple text file in your web site that inform search engine bots how to crawl and index website or web pages.
semseogurus 0 Newbie Poster
It is a file that tells a crawler how it should behave on your site.
Search engines send out web crawlers, also known as bots to websites which follow links and collect bits of information such as the number of images and meta tags.A robot.txt is a text file on the web server which tells these crawlers what they can and cannot do (although they don't have to be followed). An example would be to tell them not to index certain pages or even entire directories.
In SEO, they can be useful for stopping them attempting to visit pages which require logging in, or to stop them visiting pages which are X months old. This way, the search engines only view the latest content that you have selected.
Yes, i am agree with your answer.
almostbob 866 Retired: passive income ROCKS
note the corrections in allenhill99 & creditcards replies
robot.txt is nothing
the file must be named robots.txt
ravinder yadav 0 Newbie Poster
but where is this file attached? is it attached with index file(i mean is it in root directory of our site?)
Octet 45 Newbie Poster Featured Poster
The file goes in the root directory of your website, yes.
ravinder yadav commented: thanks +0
bhav123 0 Junior Poster in Training
A robots.txt file restricts access to your site by search engine robots that crawl the web. It is a text file that tells search engine spiders also known as search engine robots, which parts of your website they can enter and which parts they can’t.
Be a part of the DaniWeb community
We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.