My website is an ecommerce website and it has more than 500 pages, the SEMRush is only crawling 4 pages and i checked on google search conscole and all the pages there are indexed. My internal linking is also good. Help me understand why semrush is not crawling all my pages?
Recommended Answers
Jump to PostIt sounds like SEMrush might be facing restrictions while crawling your site. Check if your crawl limit is too low, ensure robots.txt isn't blocking pages, and verify your XML sitemap is updated. Also, if your site relies on JavaScript, SEMrush may struggle to crawl it properly. Try adjusting these settings …
Jump to PostAs Harry points out, my first guess would be to check robots.txt and ensure you aren't blocking any pages from SEMRush. Also make sure you aren't using a CDN or proxy like Cloudflare that is blocking it from their side.
All 7 Replies
Harry Brook 0 Newbie Poster
It sounds like SEMrush might be facing restrictions while crawling your site. Check if your crawl limit is too low, ensure robots.txt isn't blocking pages, and verify your XML sitemap is updated. Also, if your site relies on JavaScript, SEMrush may struggle to crawl it properly. Try adjusting these settings and re-run the crawl.
Dani 4,653 The Queen of DaniWeb Administrator Featured Poster Premium Member
As Harry points out, my first guess would be to check robots.txt and ensure you aren't blocking any pages from SEMRush. Also make sure you aren't using a CDN or proxy like Cloudflare that is blocking it from their side.
naps2 -12 Newbie Poster
If you are encountering a page crawling issue, you should check your robots.txt file to ensure that important pages are not blocked. You should also check Google Search Console for crawl errors or indexing issues.
Dani commented: Is this AI generated? -8
MasoodDidThat commented: As i said, all pages are indexed by google and i already checked search conscole so the idea of robots.txt file does not make sense, but anyway thanks +0
Olu_610 30 Newbie Poster
I have encountered similar issue. SEMrush appears to focus on crawling only the most prestigious pages because it operates on a 500-page limit. Using the SEMrush API alongside adjusting crawl settings will help raise the crawl depth.
Dani commented: Thanks! +34
ashleydent4u -12 Newbie Poster Banned
If Google Search Console shows your pages are indexed but SEMrush is only crawling a few, it could be due to your crawl settings or restrictions. Check your robots.txt file and meta tags to ensure SEMrush isn’t blocked from crawling your site. Also, look at your crawl budget settings in SEMrush—if it’s set too low, it won’t scan all your pages. Another issue could be your website’s server response time; slow-loading pages might not be fully crawled. Try increasing the crawl limit in SEMrush settings and re-run the audit to see if it picks up more pages.
MasoodDidThat -4 Newbie Poster
Found the solution
I tried a method where I went to the Site Audit settings on Semrush, uploaded a .txt file containing all the URLs I wanted to get indexed, and then re-ran the campaign. It worked.
jonathannweyer -3 Newbie Poster
If SEMRush is only crawling a few pages, it might be due to the crawl settings or limits in your SEMRush account. Check if there are any crawl restrictions set up and make sure your robots.txt file isn’t blocking any important pages.
Be a part of the DaniWeb community
We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.