Member Avatar for borobhaisab

Hiya,

I need to know something.
First I thought Sitemap Xml files will list all .html and .hml and .shtm and .shtml files. All pages of the website.
But now I see, Sitemap xml files also list other xml files. Check this one out for what I mean:
https://www.rocktherankings.com/sitemap_index.xml

So that means, I got to program my web crawler to go one level deep to find the site links (html files).
Question is: Does this happen more than one level deep ?
I mean does it do this ....
I go to a Sitemap xml file.
I see further xml files. I clickover to an xml file. Thus go one level deep.
I see more xml files listed. I clickover to an xml file. Thus go two level deep.

How many levels deep can a site go like this to list their html files ?
I need to know this to program my crawler how many levels it should check before giving up. Do not want to be going in an endless loop and get my crawler get into a trap.

  • 3 Contributors
  • 4 Replies
  • 936 Views
  • 3 Weeks Discussion Span
  • Latest Post Latest Post by Dani

Dani AI

Generated

A concise summary and practical checklist for crawler behavior when you encounter sitemap index files.

Many sites publish a sitemap index that points to many sitemap files instead of listing every page directly. 's point about using recursion is valid, but add a few safety measures so the crawler cannot loop or overwhelm the site: keep a visited set of sitemap URLs, rate-limit concurrent fetches, and prefer queue-based processing (BFS or DFS) rather than blindly following "levels". 's pointer to reading guides is useful for discovery steps, but the implementation details below make the crawler robust in production.

Example crawl loop (pseudo-code):

queue = [start_sitemap_url]
visited = set()

while queue:
    url = queue.pop()
    if url in visited: continue
    visited.add(url)
    xml = fetch_and_decompress(url)
    if is_sitemap_index(xml):
        for loc in xml.findall('sitemap/loc'):
            if loc not in visited: queue.append(loc)
    else:
        for loc in xml.findall('url/loc'):
            emit_page(loc)

Practical limits and rules to follow: sitemaps are constrained by the Sitemap protocol — a sitemap file must be no larger than 50MB (uncompressed) and contain no more than 50,000 URLs; a sitemap index can list up to 50,000 sitemaps. Sitemaps may be gzipped to save bandwidth. See the official Sitemaps protocol for details. (sitemaps.org)

Discovery and hosting notes: check the site’s robots.txt for Sitemap: directives (major engines honor that). Google also documents that sitemaps referenced by an index should be hosted on the same site and typically in the same or a lower directory; Search Console has submission limits to be aware of. When crawling, handle gzipped content, respect HTTP status codes (retry on transient 5xx, skip on 4xx), validate XML safely (disable external entity expansion), and log/sketch a sensible hard cap on total fetched files to avoid runaway behavior. (developers.google.com)

Recommended Answers

All 4 Replies

Member Avatar for Dani

There can be an infinite number of levels of sitemap index files pointing to sitemap index files pointing to sitemap files.

Luckily, the BJ_Crawler class you described over in your other thread already uses recursion to currently handle an unlimited depth, so you don't need to handle this on your end. It's already being taken care of for you.

Member Avatar for borobhaisab

How to cloe this thread ?

Member Avatar for Dani

You can mark it as solved by toggling the little green switch next to the reply button. I’ll go ahead and do that for you.

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.