Hi all,
For the past one week my site is not crawled by google bot . I cant understand what is the reason behind it???..Please do help me out
nissy 0 Newbie Poster
anthonydaly 16 Junior Poster in Training
Add your site on Google Webmaster Tools, there you can submit a sitemap (which you can generate for free at http://www.xml-sitemaps.com/)
After this, you should start to see some of your pages getting crawled. You can manually submit to search engines too, but you may have to wait a little bit for results to show.
Make sure that you haven't got any meta robots tags set to "noindex", as that would prevent search engines accessing your site.
PixelatedKarma 65 Junior Poster in Training Featured Poster
You didnt provide enough details for us to help.
Is your website new? If not, how old is it?
If you have a new website it could take 2-3 months for your website to get crawled, especially if you don't have a good inbound link strategy.
How often is content updated on it?
If you have an old website but you have not given it any attention in the last few months; google may have slowed the crawl rate of your website as well this is because Google looks at your website and goes "hey this content isnt being updated, why should we spend the time and resources on crawling it if content isn't updated regularly?"
How many unique visitors does your website get a day?
Using google analytics you should be able to see exactly how many people Google is recording to your website. If you aren't hitting high enough numbers; Google won't crawl you often enough, it just isnt worth their time. And if you are running a personal blog, no your mom visiting your blog 50+ times a day won't count as individual visits in Google's eyes.
How popular is your website?
I know lots of people trash on Alexa; but sites <1,000,000 tend to be crawled on a more regular basis. After all Alexa is considered a social signal by Google.
Does your website have good social signals?
Further to above, are you on the proper social media channels? Are those social media channels effective for you? The more likes, status updates, tweets, followers, etc. = the more you are making your precense known to both people and to internet crawlers. Lots of time in Internet Marketing we talk about Social Media in terms of getting people exposed to your product and service, but for a second lets focus on the technical crawler aspect.
Facebook is one of the biggest websites in the world, it gets crawled daily if not nearly hourly. If you have a website (1st tier link) which has a connected facebook page (2nd tier link) and you think of each user as a 3rd tier level link. Now each time those crawlers crawl FB, Google+, Twitter, etc.; if you have lots of followers(which means you have a larger reach and more chance of the crawlers seeing your status updates) and lots of status updates and maybe even include a link or two directly to your website in those statuses you are going to build more and more inbound links from tiers 3 to 2 which will boost the importance of crawling your tier 1 link to the crawler itself. Quantity + Quality = Good Results, Quantity + No Quality = Bad Results, No Quantity + Quality = Bad Results.
How many quality inbound links do you have? How is your internal link structure?
To add to the above question, if you don't have a good internal link structure on your website any of your buried content won't be easily indexed by crawlers. Try to operate with the internal link structure of 3; meaning you don't need to click more then 3 links from the main page to find a piece of content.
That being said you can have the world's greatest content BUT if nobody is sharing it and there are no inbound links to it, then it'll be next to impossible for crawlers to find your site let alone your super amazing content. So whether you gain reputable links through guest blogs, social media, directories, link partnerships, etc - you need atleast a few inbound links to be found. Further more the higher the quality of relevant inbound links (ie. links from websites which are crawled really often, high mR websites, etc.) the better.
Does your website validate? How many errors are there? Whats your website loading time
If your website doesn't validate through w3c or if scripts are broken this can sometimes result in fatal crawl errors which will terminate the crawler from further indexing your website past that error. This means that either that page(s) won't be indexed or they will be indexed incorrectly.
Even if your website is 100% error free; if your website takes more then a few seconds to load - a crawler could timeout before ever being able to complete the crawl. Especially if your website doesn't utilize caching. Remember, chances are like the other 99% of websites online today - you aren't worth it to Google to tie up their crawlers all day because your website is bloated.
Does your website meet SEO best practices?
This is huge and yes it covers alot of my above questions; today the web is more about being semantic then anything else and if you aren't following SEO best practices you either won't rank well, won't rank as well, or won't rank at all.
Ohh also on a final note; sometimes your website can be indexed but it won't show up for a week+ in Google webmaster tools. If 66% or more of your website is showing up as indexed; then I wouldn't worry about it as sometimes it just takes awhile for it to show up correctly in the tool itself.
mindstreak 6 Junior Poster
Check if sitemap is intergrated for the website.Add your website in google webmaster and check for google site verification.You can also use the fetch option in google webmaster for your webmaster.Most importantly, make sure you wait for sometime since crawling does not occur frequently.
digvijaybook 0 Newbie Poster
I also faced these issues many times, but there no need to worry becouse sometimes google takes time. If are continues submitting url in Google webmaster.
Be a part of the DaniWeb community
We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.