Member Avatar for louie540

Hey.

I have a website that is user submitted which has a repot button and the Google crawler keeps spamming that button. I am aware of robots.txt but it seems that nothing is working to stop Google from clicking it. So instead, I'd like to assign the link an ID and hide it from Google's IP address.

Can this be done?

Thanks.

  • 5 Contributors
  • 4 Replies
  • 305 Views
  • 12 Hours Discussion Span
  • Latest Post Latest Post by Igal-Incapsula
Member Avatar for LastMitch

@louie540

I have a website that is user submitted which has a repot button and the Google crawler keeps spamming that button. I am aware of robots.txt but it seems that nothing is working to stop Google from clicking it. So instead, I'd like to assign the link an ID and hide it from Google's IP address.

This question is more suited for Internet Marketing Section rather than PHP section. I understand what you are mention about the robot.txt but if you have a code in PHP than it is OK to post it here. Otherwise post it there. If you have a PHP code just post the code in the PHP section.

Member Avatar for leviathan185

Google will pry into your life not matter how much you try and stop them.... Just sayin

Member Avatar for Echo89

You could achieve this with the user agent maybe?

Member Avatar for Igal-Incapsula

Hi
Googlebot will ALWAYS comply with robots.txt directives.

I think you may be suffering from Googlebot impersonation attempt.

Here is a link to learn more about that:
http://www.incapsula.com/the-incapsula-blog/item/369-was-that-really-a-google-bot-crawling-my-site

My advise is:

  • Use Google WMT and do a manual "Fetch" for the button URL. If you succesed than your robots.txt is not really blocking it and you'll need to modify it accordingly.

  • If "Fetch" fails (meaning your robots.txt settings are fine) than you should get the user-agent and IP info of this "Googlebot" and verify them on http://www.Botopedia.org

If user-agent data or IPs won't match you'll know that something is wrong. In this case the visiting IP should be blocked because this is either a spam attempt or a vuln. scan performed for future hacking purposes. (somethinkg like Havij for exmple)

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.