Summary
When the sitemap (or the robots.txt used to discover it) is fetched through the default proxy and the target returns an anti-bot block — typically HTTP 403 — the run fails with "No valid sitemaps were discovered" even though a sitemap actually exists. Propose falling back to Apify Unblocker (or a residential proxy with a browser-like fingerprint) for sitemap/robots.txt retrieval when a block is detected.
Evidence
From the 500-domain test (Apr–Jun 2026), 101 runs FAILED. Classifying their logs:
| Reason |
Count |
% |
| No sitemap genuinely absent |
63 |
62.4% |
| Blocked / anti-bot (sitemap likely exists) |
21 |
20.8% |
| DNS / domain unresolvable |
15 |
14.9% |
| TLS/SSL error |
1 |
1.0% |
| Server 5xx while fetching |
1 |
1.0% |
~1 in 5 failures were false negatives — real, sitemap-having sites returning 403 to the sitemap/robots fetch. Examples: help.shopify.com, coupang.com, tableau.com, mayoclinic.org, docs.redhat.com, qconcursos.com.
The current logs already show the pattern:
WARN Sitemap discovery through proxy failed or returned no sitemaps. Retrying once without proxy.
ERROR [Status message]: No valid sitemaps were discovered from the provided startUrls.
The "retry without proxy" step is not enough — these hosts block datacenter IPs and basic fingerprints outright.
Proposed fix
- Detect anti-bot blocks (HTTP 401/403/429; Cloudflare / "just a moment" / captcha markers) on sitemap and robots.txt fetches.
- On block, retry the fetch through Apify Unblocker (or residential proxy with a browser-like fingerprint) before giving up.
- Escalate to Unblocker only on block (cost control) — keep the fast default-proxy path first.
Expected impact
Recovers ~20% of current sitemap failures — real sites we drop today — and does so on exactly the high-value large/popular domains most likely to sit behind anti-bot.
🤖 Filed from a data-driven analysis with Claude Code, based on a 500-domain Sitemap Extractor test (Apr–Jun 2026).
Summary
When the sitemap (or the robots.txt used to discover it) is fetched through the default proxy and the target returns an anti-bot block — typically HTTP 403 — the run fails with "No valid sitemaps were discovered" even though a sitemap actually exists. Propose falling back to Apify Unblocker (or a residential proxy with a browser-like fingerprint) for sitemap/robots.txt retrieval when a block is detected.
Evidence
From the 500-domain test (Apr–Jun 2026), 101 runs FAILED. Classifying their logs:
~1 in 5 failures were false negatives — real, sitemap-having sites returning 403 to the sitemap/robots fetch. Examples:
help.shopify.com,coupang.com,tableau.com,mayoclinic.org,docs.redhat.com,qconcursos.com.The current logs already show the pattern:
The "retry without proxy" step is not enough — these hosts block datacenter IPs and basic fingerprints outright.
Proposed fix
Expected impact
Recovers ~20% of current sitemap failures — real sites we drop today — and does so on exactly the high-value large/popular domains most likely to sit behind anti-bot.
🤖 Filed from a data-driven analysis with Claude Code, based on a 500-domain Sitemap Extractor test (Apr–Jun 2026).