Member Avatar for davy_yg

What is robot.txt? Is it necessary to have robot.txt for a webpage? Does it effect SEO at all?

Thanks.

  • 3 Contributors
  • 2 Replies
  • 95 Views
  • 8 Hours Discussion Span
  • Latest Post Latest Post by Jessfly
Member Avatar for JorgeM

There is quite a bit of information out there regarding the use of the robot.txt file. If you as the web site owner, want to give specific instructions to web spiders/robots, you would place a text file called robots.txt in the root of the web site directory. This text file should include instructions in a specific format that instruct the robot on whether to allow or deny it access to folders and/or files.

So, if you have a certain directory or files that you do not want to be indexed by a search engine, you should definately include this file.

If you get an account with Google Webmasters, you can use their online tool to see the effects of a robots.txt file on your site.

Edited by JorgeM
Member Avatar for Jessfly

You can allow or disallow search engines to crawl your site. Robot.txt is helpful for crawling a site by SE spiders.

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.