Hi,
My website is being crawled by a crawler by name yandex.ru almost 24 hours a day. How can I stop it from crawling my website. My website is USA based. Hence, I don't want this crawler to crawl my website all the time.
Any advice please.
Thanks in advance
Create a robots.txt file with the following content. This will prevent Yandex from crawling your site.
User-agent: Yandex
Disallow: /
User-agent: Yandex/1.01.001
Disallow: /
edit your robots.txt file
Thanks to all the post, esp., to inspirohost. Now I have implemented it. Now that spider is not crawling my website.
Thanks again
Very useful information, thanks.
Thanks to all the post, esp., to inspirohost. Now I have implemented it. Now that spider is not crawling my website.
Thanks again
No problem at all, glad to help :)
write your robots.txt file and stop it.
You can also do it on per page basis..put meta robot tag in head section of your page...
<META NAME="ROBOTS" CONTENT="INDEX, NOFOLLOW">
Is there a drawback to restricting Yandex? i.e. Is anyone seeing quality traffic from the site?
you can use robot.txt file to stop crawling particular pages.
You could block them in your robots.txt file if such crawlers obey the rules. If not, I would recommend you to block them directly via IP blocker inside your hosting control panel.
Have a nice day,
Hi,
My website is being crawled by a crawler by name yandex.ru almost 24 hours a day. How can I stop it from crawling my website. My website is USA based. Hence, I don't want this crawler to crawl my website all the time.
Any advice please.
Thanks in advance
nice! your site is continuly indexing.what strategy you are using ? good knowledge i got here.
thanks
We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.