what is robots.txt?
please define robot.txt what is it and how does it works?
smith09 -1 Light Poster
InsightsDigital 57 Posting Virtuoso
Robots and spiders crawl your site. The file is there to give these spiders instructions once they visit your site. http://www.seoconsultants.com/robots-text-file/
shewag 0 Newbie Poster
Hi friends,
Meaning for Robots.txt:
Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines .
you are interested to learn more about it, visit http://www.robotstxt.org/ or you can go straight to the Standard for Robot Exclusion.
Shewag
infinique 0 Junior Poster
Robots.txt file is a file that gives the Search Engine crawlers the right instructions. E.g such as where is the sitemap.xml file and where to not follow.
eysiojo23 0 Light Poster
Robots.txt file allows or prevents Search engine crawlers to enter your site. It is like a instruction manual or a map for the crawlers to know where to crawl.
jhonsadins 0 Newbie Poster
What they're telling you is that the internet archive did not archive the site due to a preference set in that site’s robot.txt file.
The robot.txt file is checked by spiders, like the way back machine, to determine whether or not the site’s owner wants that particular spider to index their site.
infinique 0 Junior Poster
Robots.txt is a file that directs and gives instructions to the search engine spiders on what to follow and what not.
eysiojo23 0 Light Poster
Karol said it all. there are so many members here that will surely help.
mackone 55 Posting Pro
robots.txt basically tells search engine spiders not to index certain areas of your site. You can protect private areas of sites from being visible to search engines, and thus to everyone.
treats 0 Newbie Poster
Robots.txt is a text file you put on your site to tell search robots which pages you would like them not to visit.
redesignunit -4 Posting Pro in Training
Robot.txt can help to get index ur site and crawl your site page
AirForceOne 9 Posting Pro in Training
Robots.txt is used for giving instructions about their site to web robots. It resides under our site root directory. In my own experiences, it might be also a leak for our sites since it is only a plain text file and would be found easily by hackers.
All the best,
telugucinimalu 0 Newbie Poster
robots.txt is a text file which can be used to restrict web robots to accessing your web site only in ways of which you approve.
The robots.txt file is a simple text file (no HTML), that must be placed in your root directory, for example:
iflexion 0 Newbie Poster
To make a robots.txt file you don't need any special knowledge. Just an ordinary Notepad application. Try to google this term and you'll find lots of examples with an explanation of what each line means. And you'll easily compose one for yourself.
sanasahil 0 Light Poster
robot.txt is a text file(not HTML) that tell search engine bot what to follow and what not to!
Kilandara 0 Newbie Poster
If you do not know what robots.txt are, chances are you are better off not specifying them in your code. As you could get into trouble by not having your site pages crawled... and I mean EVERY page
jay 11 8 The Dude Abides Team Colleague
If you do not know what robots.txt are, chances are you are better off not specifying them in your code. As you could get into trouble by not having your site pages crawled... and I mean EVERY page
SE's don't need to crawl every page. For example, do they need to crawl your website's terms of service or privacy policy? Or how about a co-branded promo page that you know is only going to be 'live' for an extended period of time?
Not every page needs to be indexed and crawled by the Search Engines. In fact, it would be great for the WWW if webmasters were more selective of the pages they allowed the SE's to access. There would be a lot less glut out there. Think of all the rack space Google and Bing could save! Everyone... Go Green.. Use Robots.txt! ;)
mystryworld 0 Light Poster
robots.txt is basically used to tell the crawler where to crawl and which section you don't need to be crawled. While optimizing your site you can keep this text file under your root directory.
induswebi 0 Newbie Poster
The information was too quite well.
joelchrist -9 Posting Whiz
Robot text file is like set of instruction to search engine spider about crawling site data.
babadirect 0 Newbie Poster
i was getting into the same question... thanks all who replied
Prosquote 0 Light Poster
BretlenHock 0 Newbie Poster
The robots.txt is a standard which was developed in 1994, when large-scale web indexing became popular; indexers such as Lycos and AltaVista used it
Prosquote 0 Light Poster
robots.txt is mainly use for leave those pages which you not want to read by Google...
healthdiettips 0 Newbie Poster
Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit.The location of robots.txt is very important. It must be in the main directory because otherwise search engines will not be able to find it.
Grahamjenny 0 Newbie Poster
Robots.txt File should be included so that the search engine bots don't get 404 errors when they look for it.
Just include the following 2 lines and drop it in the root.
User-agent: *
Disallow:
infowavehost 0 Newbie Poster
A text file placed in the root directory of a Web site that prohibits search engine spiders from indexing all or specific pages of the site.
findmemak 0 Newbie Poster
The robots.txt file is a set of instructions for visiting robots (spiders) that index the content of your web site pages. For those spiders that obey the file, it provides a map for what they can, and cannot index.
Definition of the above robots.txt file:
User-agent: *
The asterisk (*) or wildcard represents a special value and means any robot.
Disallow:
The Disallow: line without a / (forward slash) tells the robots that they can index the entire site.
Example Code for robots.txt file:
User-agent: *
Disallow: /private/
Disallow: /images-saved/
nazar22 0 Light Poster
Hello All
Robots.txt basically tells search engine spiders not to index certain areas of your site.
Code:
<meta name="robots" content="index,follow" />
franklin4005 0 Light Poster
its a file.. when you put on your site after that Google will be crawler you site..
-----------------
Be a part of the DaniWeb community
We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.