How does one go about creating a robots.txt file and placing it in a web site? I hear a lot of talk about robots.txt and how important it is for SEO.

Upload robot.txt to your root directory.

Well, its a simple text file.

User-agent: *
Allow: /blog
Disallow: /mail

This is the Robots.txt of my website, user-agents determine the bot, i.e., if its google or bing or whatever. Allow tabs tells which part of the domain to be indexed by the SEs. Disallo part orders the bot to not crawl the mail directory.

a. with a text editor
b. with an ftp client

[voice=porky]that's all folks[/voice] gotta make a joke sometime

robots.txt is just a text file, notepad will do it, the content is simple there are dozens of short tuts on what it needs to contain to accomplish the litany it is capable of
and being a text file is real easy to reset if something goes awry

Oh lol Bob, I didn't knew that you could edit with a text file.. lol don't you think he needs access to the file first before editing it? Just rearrange the options lol :)

Oh lol Bob, I didn't knew that you could edit with a text file.. lol don't you think he needs access to the file first before editing it? Just rearrange the options lol :)

a: development is done locally and uploaded to the server
no, the op would not require the ftp client first,
or
b: the questions were asked by the op sequentially, and answered in the same sequence

speaks to the skill, or lack thereof, you are bringing to the clients you seek in your signaturego away intellectual amoeba

                User-agent: *
                Disallow:  /

                           Now, the search engine don't crawl and index your pages.

You add this code to your robots.txt file. Your robots.txt file should be present in root directory of your websites

commented: total fail... -3

Guru, if he write your code, SE bots never index his page. Please dont misguide him.

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.