Member Avatar for jennia1686

How does one go about creating a robots.txt file and placing it in a web site? I hear a lot of talk about robots.txt and how important it is for SEO.

  • 6 Contributors
  • 7 Replies
  • 186 Views
  • 3 Years Discussion Span
  • Latest Post Latest Post by Ketul_1
Member Avatar for takeonepicture

Upload robot.txt to your root directory.

Member Avatar for Steven.Matrix

Well, its a simple text file.

User-agent: *
Allow: /blog
Disallow: /mail

This is the Robots.txt of my website, user-agents determine the bot, i.e., if its google or bing or whatever. Allow tabs tells which part of the domain to be indexed by the SEs. Disallo part orders the bot to not crawl the mail directory.

Member Avatar for almostbob

a. with a text editor
b. with an ftp client

[voice=porky]that's all folks[/voice] gotta make a joke sometime

robots.txt is just a text file, notepad will do it, the content is simple there are dozens of short tuts on what it needs to contain to accomplish the litany it is capable of
and being a text file is real easy to reset if something goes awry

Member Avatar for Steven.Matrix

Oh lol Bob, I didn't knew that you could edit with a text file.. lol don't you think he needs access to the file first before editing it? Just rearrange the options lol :)

Member Avatar for almostbob

Oh lol Bob, I didn't knew that you could edit with a text file.. lol don't you think he needs access to the file first before editing it? Just rearrange the options lol :)

a: development is done locally and uploaded to the server
no, the op would not require the ftp client first,
or
b: the questions were asked by the op sequentially, and answered in the same sequence

speaks to the skill, or lack thereof, you are bringing to the clients you seek in your signaturego away intellectual amoeba

Edited by almostbob because: n/a
Member Avatar for guruparthi 
                User-agent: *
                Disallow:  /

                           Now, the search engine don't crawl and index your pages.

You add this code to your robots.txt file. Your robots.txt file should be present in root directory of your websites

commented: total fail... -3
Member Avatar for Ketul_1

Guru, if he write your code, SE bots never index his page. Please dont misguide him.

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.