Sitemap Extractor: overlap sitemap download with crawling (enqueue URLs as they are parsed, don't block on full enumeration)

[nicklamonov]

Summary

When useSitemaps is enabled, the scrapers currently enumerate the entire sitemap tree before the crawler processes any page. For large sites whose sitemap index references hundreds of child sitemaps, this serial "download-all-sitemaps-first" phase dominates run time and frequently exhausts the run timeout before real crawling even begins.

Proposal: stream sitemap URLs into the request queue as each child sitemap is parsed, and start the crawler concurrently, so sitemap I/O overlaps with page processing.

Evidence

From a large-scale test — 500 domains (250 most-popular + 250 random), run through a Crawlee-based Sitemap Extractor over an Apr–Jun 2026 usage window:

• Top-250 popular sites averaged ~2,355 sitemap URLs/domain vs ~290 for the random sample (~8× more).
• 83 of 500 runs hit the 1-hour timeout, disproportionately the large sites — they spent the bulk of the run downloading/parsing sitemaps and never got through the enqueued links. Their discovered-page counts are truncated as a result.

Root cause

Crawlee's Sitemap.load() (the standard path) collects every URL from the whole sitemap tree — following the sitemap index and all child sitemaps — into a single urls array, and only returns once complete. The actor then calls crawler.addRequests(sitemap.urls). So wall-clock is effectively sitemap_download_time + crawl_time, fully serial.

(Confirmed against the Crawlee Sitemap API, which exposes load() / fromXmlString() / tryCommonNames() but no incremental/streaming consumption — all URLs surface only after the full tree is loaded.)

Proposed fix

Turn sitemap ingestion into a producer/consumer that overlaps with crawling:

1. Fetch the sitemap index first to obtain the list of child-sitemap URLs.
2. Download child sitemaps with a bounded-concurrency pool (e.g. 10–20), parsing each with Sitemap.fromXmlString() (or a streaming parser) rather than the monolithic load().
3. As each child sitemap resolves, immediately await requestQueue.addRequests(batch) — don't wait for the rest.
4. Start the crawler concurrently (don't await full enumeration). Crawlee's autoscaled pool pulls URLs as soon as they land in the queue.

Result: wall-clock drops from sitemap_time + crawl_time to roughly max(sitemap_time, crawl_time), and large sites make real crawl progress before hitting the timeout.

Notes

• Cap per-host sitemap concurrency to avoid triggering anti-bot (see the related Unblocker issue).
• Complements a separate sitemap download-optimization issue (streaming XML, gzip, keep-alive, incremental skip).

---

🤖 Filed from a data-driven analysis with Claude Code, based on a 500-domain Sitemap Extractor test (Apr–Jun 2026).

[nicklamonov]

Example run: https://console.apify.com/actors/tasks/9xGaItNDKs6ppinEz/runs/Om2BVDl9HnXe4BYc7#log

[nikitachapovskii-dev]

Sitemap Extractor doesn't enumerate the whole tree, then crawl. Page URLs are enqueued incrementally (via parseSitemap, batches of 25) as each sitemap is parsed, and pages are pulled from the same queue right away.
As I pointed out on a meeting yesterday, there is discovery step to find top level sitemaps which is not overlapping but it's too tiny to have significant impact (30s timeout for whole process)
In the timed-out runs under your account the enumeration wasn't the problem, individual sitemap requests were dying on the 60s requestHandler timeout (slow hosts / large bodies), made much worse by concurrency being stuck at 1 (CPU overloaded -> desiredConcurrency: 1). Other cases are encoding problem

cc @nicklamonov @ruocco-l

[nicklamonov]

There might be some confusion by Claude about specific configuration causing the problem.

If the root cause is rather in timeouts and concurrency, let's try to address this first.

But at the same time, if we take the example run attached, there is nothing there about overloading resources or timed out requests for sitemaps. They looks like they were parsed correctly and added to request queue for head requests, but that queue was never started to be processed within the run (because it was still being filled by coming sitemaps).
Or am I reading it wrong?

[nikitachapovskii-dev]

if we take the example run attached, there is nothing there about overloading resources or timed out requests for sitemaps

my bad, forgot to mention, it's also bad decoding case
https://console.apify.com/actors/runs/InCICnhlx73KV8HmV#log -same input
The sitemaps parsed fine and the pages were enqueued, they just all die on that body read

UPD: The tree was being processed until 12:08:30 (started at 12:03) but we started to get errors from 12:07. The 4 min before pages is real but a rare case: a huge tree (autopush fanned into ~280 per-country sitemaps) that filled the slots ahead of the pages and changing that FIFO order is not probably what we want.

[nicklamonov]

changing that FIFO order is not probably what we want

Isn't this the whole point of the change? )
I mean not to change the stack logic, but rather make "2 stacks/queues" in some way?

If I understand well, the current setup is:

1. Parse sitemap index and enqueue child sitemaps
2. Go through child sitemaps and enqueue discovered links at the bottom of the queue
3. When sitemaps are fully processed, go through pages

One proposed version of the flow (if we keep only 1 queue):

1. Parse sitemap index and enqueue one child sitemap
2. Go through this child sitemap and enqueue pages from it, process these pages.
3. Only then parse another child sitemap, enqueue pages from it, go through them. Repeat.

Alternative (if possible and not overly complex):

1. Parse sitemap index and enqueue child sitemaps to "sitemaps queue".
2. In parallel start enqueuing and processing pages discovered in main request queue.

[nikitachapovskii-dev]

I mean not to change the stack logic, but rather make "2 stacks/queues" in some way?

If I get it correctly the original idea was to merge the tree-traversal phase into processing, and they are already merged: it's a single queue where sitemap parsing and page processing overlap (in that run pages ran from 12:07 while tree finished at 12:08:30). So it's not "when first step is fully processed, go through pages"

Page requests can be enqueued with forefront. I remember we discussed this with @ruocco-l but decided to not use forefront. Maybe we should get back to that topic.

A real two-queue split is doable (WCC does exactly that, a separate sitemap crawler feeding the main queue), but that's a second pool competing for the same resources instead of one autoscaled pool, in WCC the sitemap pool is pinned to maxConcurrency: 1, so it doesn't fan out and fight the main pool for resources so much

[nicklamonov]

Page requests can be enqueued with forefront. I remember we discussed this with @ruocco-l but decided to not use forefront.

Yeah, I also don't think it would be efficient.

A real two-queue split is doable (WCC does exactly that, a separate sitemap crawler feeding the main queue), but that's a second pool competing for the same resources instead of one autoscaled pool, in WCC the sitemap pool is pinned to maxConcurrency: 1, so it doesn't fan out and fight the main pool for resources so much

So if we cap that second (sitemap) pool to low concurrency (maybe not 1, but 2-3, so it can still process multiple sitemaps in parallel), it may not impact the main queue efficiency?

What would you recommend?

[nikitachapovskii-dev]

So if we cap that second (sitemap) pool to low concurrency (maybe not 1, but 2-3, so it can still process multiple sitemaps in parallel), it may not impact the main queue efficiency?

Even now concurrency gets throttled by CPU in some cases, a second independent pool competing for the same CPU would push on that more.
WCC can pin its sitemap pool to low concurrency because its main pool does the heavy work (per-page rendering, Playwright by default plus enqueues found links), so the light sitemap side never holds it back. Here it's usually the opposite: the sitemap parsing is itself the CPU-bound part, with no heavier stage to hide behind, so capping that pool low would make more time per item in output.

But why we don't want to use forefront? I mean we could try at least it seems like the safest option

[nikitachapovskii-dev]

Ok looked into forefront, it's not a great option here.
When there are forefront requests in the queue, RQv2 keeps re-listing and unlocking the head more often so it's an extra CPU/queue