Member Avatar for smithjohn1

Define Robot.txt and Spider.Also discuss about difference between them.

  • 2 Contributors
  • 1 Reply
  • 63 Views
  • 8 Hours Discussion Span
  • Latest Post Latest Post by LastMitch
Member Avatar for LastMitch

Define Robot.txt and Spider.Also discuss about difference between them.

Robot.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit.

Spiders will finds new webpages and submits to the database. The reason it's called a spider because it crawls over the Web.

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.