Hi,
I have a quick question! I have implemeted robots.txt to disallow some of my pages! How long it will take to do this? whether i can get all my pages OUT OF INDEX all at a time or it will happen pages by pages. I'm confused
Ideas are welcome
harininet 0 Newbie Poster
stymiee 111 He's No Good To Me Dead Team Colleague
It depends on your site. Sites that are crawled more frequently will see results faster then sites crawled less frequently. If you want pages removed from the SERPs quickly robots.txt is not your best option. Use a removal request instead.
nytrokiss 0 Light Poster
when the seach engine crawls your site the first file it hits is robots.txt and if it sees to disallow any files it won't index them which means they should go off there index!
webmasterran 1 Junior Poster in Training
when the bot crawls thats the time when it'll see what is allowed and whats not allowed and make sure u use opening slash / and closing slash /
/images/
something like that when not allowing the bot to crawl with the right syntax =)
Candy_ME 0 Junior Poster in Training
Yes, I agree with webmasterran. I too use robot,txt for my sites to say the search engines not to crawl my page.
Be a part of the DaniWeb community
We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.