Member Avatar for webguru_1

What's the significance of Robots.txt file in a website?

  • 6 Contributors
  • 5 Replies
  • 244 Views
  • 1 Month Discussion Span
  • Latest Post Latest Post by sandy_5
Member Avatar for Dani

To tell bots (such as Googlebot) and other spiders and crawlers what parts of your site to avoid.

Member Avatar for pty

The irony is palpable.

commented: Webguru asks this question? +6
Member Avatar for Vikash_12

The robots.txt file is used in a website to tell web robots which pages to crawl and index and which not.

Member Avatar for agustina

Robots.txt files are beneficial to you for search engines to ignore the duplicate pages on your website.

Member Avatar for sandy_5

The robots exclusion protocol (REP), or robots.txt is a text file webmasters create to instruct robots how to crawl and index pages on their website.

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.