Member Avatar for nissy

Dear All,

I want google bot to visit all my web pages. What type of robot.txt code should be used to allow google bot visit all my webpages?

  • 5 Contributors
  • 6 Replies
  • 245 Views
  • 1 Day Discussion Span
  • Latest Post Latest Post by DistantGalaxy
Member Avatar for Dani

You don't need a robots.txt file if you don't want to disallow robots from anything. However, it can't hurt to create one to specify your sitemap file:

Sitemap: http://URL-TO-SITEMAP-FILE
Member Avatar for janeleonard

First check your robots.txt file and make sure you have the command:
User-agent:*
Allow:/

Member Avatar for DistantGalaxy

That would override any disallow directives you may have defined, but only for those web crawlers that observe the Allow directive. Others could still be blocked.

Edited by DistantGalaxy
Member Avatar for Dani

That's why I say it's better to just be blank.

Member Avatar for Deepanshu_1

You need a robots.txt file only if your site includes content that you don't want Google or other search engines to index. To let Google index your entire site, don't make a robots.txt file (not even an empty one).

Member Avatar for DistantGalaxy

Not even an empty one? But what if the site contains content that is not discoverable through the normal link crawling process?

Providing a minimal robots exclusion file with just a sitemap reference, as Dani first suggested, will help to ensure all pages are crawled. Even the ones that aren't linked.

I realize the OP is only asking about Google. Yes, it's possible to notify Google about a sitemap though their Webmaster Tool, but generally speaking it's more practical to use the robots exclusion file. That way you only need to specify it once and every crawler will have the opportunity to discover it.

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.