Member Avatar for jiten702

Hi all......

As it is known to all that robots.txt file is use to specify how search engines should crawl your site's content.....but how much it is important to have it? Does it help in SEO?

  • 6 Contributors
  • 5 Replies
  • 199 Views
  • 1 Year Discussion Span
  • Latest Post Latest Post by devisrimari1
Member Avatar for InsightsDigital

I personally never had such a file and I have not seen an impact on my site. Have others?

Member Avatar for canadafred

The robot.txt file is kind of obsolete, or at least fairly redundant, if you use XML Sitemaps.

Member Avatar for Mark121

Hi jiten,
When a search engine crawler or spiders visit your site then at first they search Robots.txt file.This file tells the search engine spider, which Web pages of your site should be indexed and which Web pages should be ignored. If there is a Robots.txt file, then by default it will try to crawl all the pages,then it will ensure that none of the pages are skipped.

Member Avatar for almostbob

With Help of the Robots.txt you can guide the Search Engine which page you want to crawl and not crawl.

Geez Dave,
its more than a year old let the thread DIE

Member Avatar for devisrimari1

hi,
Robots.txt is a text file you put on your site to tell search robots which pages you would like them not to visit.

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.