In Google Search Console, I see a giant spike in "Other agent type" in crawl stats a few days ago. We don't use Google Ads (haven't for many years), and didn't add any large amount of content recently that I could imagine would constitute Google AdSense crawl. What could this be attributed to?
Dani 4,653 The Queen of DaniWeb Administrator Featured Poster Premium Member
bijutoha 76 Junior Poster
If you’re not running Google Ads or producing a large volume of new content, chances are it’s not AdsBot or something new being discovered. Unless you're experiencing significant issues with your site, it’s usually nothing to stress over and could simply be background activity from Google.
I honestly think sometimes Google's little robots are on some sort of secret mission, checking things out or finding new ways to explore the web. They don’t always let us know what they’re up to!
Or you could see it as a cool job by Google’s bots: Maybe they’re searching for something super specific in your little corner of the internet that only those curious bots know about.
So really, there’s probably no reason to freak out. It simply means that Google sent some of its less common, specialised bots to see what’s going on in your space. They're just doing their thing to help people discover your corner of the internet!
Dani 4,653 The Queen of DaniWeb Administrator Featured Poster Premium Member
Unless you're experiencing significant issues with your site
Oh, but I am experiencing significant issues with my site. I also just noticed that within 72 hours of the spike, 80% of our URLs have been deindexed by Google. This is the first time this has ever happened in our 23 years.
bijutoha 76 Junior Poster
Unless you're experiencing significant issues with your site
Oh, but I am experiencing significant issues with my site. I also just noticed that within 72 hours of the spike, 80% of our URLs have been deindexed by Google. This is the first time this has ever happened in our 23 years.
Oh wow, Dani, that 80% deindexing changes things – that's a massive issue and sounds connected to that "Other agent type" spike. For a site of your age, my first thought is usually a manual action or a security issue from Google. Have you had a chance to check your GSC under Manual actions and Security issues yet? That's usually where Google will tell you directly if something severe happened. Please keep us posted; this needs a thorough review.
Dani 4,653 The Queen of DaniWeb Administrator Featured Poster Premium Member
No manual actions or security issues.
bijutoha 76 Junior Poster
No manual actions or security issues.
Okay, Dani, that's genuinely good news about no manual actions or security issues, at least that rules out the scariest stuff! But it makes the 80% deindexing even more puzzling.
Since it's not a penalty, my mind immediately jumps to some kind of technical configuration problem that might have happened on your end, possibly right around that Other agent type crawl spike.
You should thoroughly review your robots.txt file and check for any noindex tags that may have been added. Additionally, could you please check your server logs for any sudden increase in 404 errors or server errors? That rapid deindexing suggests that Google is being blocked or instructed not to index a significant portion of your site.
Dani 4,653 The Queen of DaniWeb Administrator Featured Poster Premium Member
Yesterday I ran a Screaming Frog SEO spider crawl of my site as a Googlebot user agent to see if there was some reason that Google was being blocked, and nothing unusual showed up.
Google crawl stats in search console don’t indicate anything out of the ordinary such as 4xx or 5xx errors.
I posted yesterday in the Google Webmaster Central forums and encountered someone else who posted that the exact same thing happened to them on the same exact day.
Be a part of the DaniWeb community
We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.