My website is an ecommerce website and it has more than 500 pages, the SEMRush is only crawling 4 pages and i checked on google search conscole and all the pages there are indexed. My internal linking is also good. Help me understand why semrush is not crawling all my pages?
Harry Brook 0 Newbie Poster
It sounds like SEMrush might be facing restrictions while crawling your site. Check if your crawl limit is too low, ensure robots.txt isn't blocking pages, and verify your XML sitemap is updated. Also, if your site relies on JavaScript, SEMrush may struggle to crawl it properly. Try adjusting these settings and re-run the crawl.
Dani 4,402 The Queen of DaniWeb Administrator Featured Poster Premium Member
As Harry points out, my first guess would be to check robots.txt and ensure you aren't blocking any pages from SEMRush. Also make sure you aren't using a CDN or proxy like Cloudflare that is blocking it from their side.
Be a part of the DaniWeb community
We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.