Those who are still confused about robots.txt.
Here is a link from mattcutts.
http://www.mattcutts.com/blog/robots-txt-remove-url/
stephen186 4 Junior Poster in Training
ayoungpretender -7 Junior Poster
An XML Sitemap (upper-case) file, which you can submit through Google's Webmaster Tools, makes it easier for Google to discover the pages on your site.
Using a Sitemap file is also one way (though not guaranteed) to tell Google which version of a URL you'd prefer as the canonical one.
mahgobb 0 Junior Poster
Those who are still confused about robots.txt.
Here is a link from mattcutts.
http://www.mattcutts.com/blog/robots-txt-remove-url/
Thanks for the sharing of this link, indeed, it clear me some doubts!
Be a part of the DaniWeb community
We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.