How important are robots.txt file?
bellasteward 0 Newbie Poster
surindersharma 0 Junior Poster
Robots.txt is used to guide search engine through your website.
if you wish that a specific folder/webpages should not be crawled by an particular search engine. then robots.txt can be helpful for you.Robots.txt can control the excess of any search engine through out your complete website.
bellasteward 0 Newbie Poster
Thanks surindersharma for replying to my post
stephaniemcgrat 0 Newbie Poster
Robots.txt file controls crawler behavior. I am copy - pasting some lines from this article I found on Redalkemi site. It hope it is useful to you
Here are some tips on how to use robots.txt file -
1. The robots.txt file is always named in all lowercase (e.g. Robots.txt or robots.Txt is incorrect)
2. The robots.txt file is an exclusion file meant for search engine robot reference and not obligatory for a website to function. An empty or absent file simply means that all robots are welcome to index any part of the website.
bellasteward 0 Newbie Poster
Hi Stephaniemcgrat..........thanks for the nice article. Its really very nicely explained and tips given in this article on redalkemi site, is really excellent
MidiMagic 579 Nearly a Senior Poster
Note that some search engines can not search certain sites unless the robots.txt file is present.
Be a part of the DaniWeb community
We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.